1,282
Views
45
CrossRef citations to date
0
Altmetric
Comprehensive Review

Automatic diet monitoring: a review of computer vision and wearable sensor-based methods

ORCID Icon, ORCID Icon, , ORCID Icon, ORCID Icon & ORCID Icon
Pages 656-670 | Received 02 Dec 2016, Accepted 15 Jan 2017, Published online: 31 Jan 2017

References

  • Almaghrabi R, Villalobos G, Pouladzadeh P, Shirmohammadi S. 2012. A novel method for measuring nutrition intake based on food image. IEEE International Instrumentation and Measurement Technology Conference (I2MTC 2012); 2012 May 13–16; Graz, Austria. IEEE Computer Society; p. 366–370.
  • Amft O. 2010. A wearable earpad sensor for chewing monitoring. IEEE Sensors 2010 Conference; 2010 Nov 1–4; Waikoloa, HI. IEEE Computer Society; p. 222–227.
  • Amft O, Junker H, Troster G. 2005. Detection of eating and drinking arm gestures using inertial body-worn sensors. Ninth IEEE International Symposium on Wearable Computers; 2005 Oct 18–21; Washington, DC. IEEE Computer Society; p. 160–163.
  • Amft O, Troster G. 2006. Methods for detection and classification of normal swallowing from muscle activation and sound. 2006 Pervasive Health Conference and Workshops; 2006 Nov 29–Dec 1; Innsbruck, Austria. IEEE Computer Society; p. 1–10.
  • Anthimopoulos MM, Gianola L, Scarnato L, Diem P, Mougiakakou SG. 2014. A food recognition system for diabetic patients based on an optimized bag-of-features model. IEEE J Biomed Health Inform. 18:1261–1271.
  • Bergh C, Brodin U, Lindberg G, Södersten P. 2002. Randomized controlled trial of a treatment for anorexia and bulimia nervosa. Proc Natl Acad Sci USA. 99:9486–9491.
  • Bossard L, Guillaumin M, Van Gool L. 2014. Food-101-mining discriminative components with random forests. Computer Vision-ECCV 2014 Workshops; 2014 Sept 6–7; Zurich, Switzerland. Switzerland: Springer.
  • Chen M, Dhingra K, Wu W, Yang L, Sukthankar R, Yang J. 2009. PFID: Pittsburgh fast-food image dataset. 16th IEEE International Conference on Image Processing (ICIP 2009); 2009 Nov 7–10; Cairo, Egypt. IEEE Computer Society; p. 289–292.
  • Christodoulidis S, Anthimopoulos M, Mougiakakou S. 2015. Food recognition for dietary assessment using deep convolutional neural networks. New trends in image analysis and processing-ICIAP 2015 Workshops. Amsterdam: Springer; p. 458–465.
  • Cortes C, Vapnik V. 1995. Support-vector networks. Mach Learn. 20:273–297.
  • Dehais J, Anthimopoulos M, Mougiakakou S. 2015. Dish detection and segmentation for dietary assessment on smartphones. New trends in image analysis and processing-ICIAP 2015 Workshops. Amsterdam: Springer; p. 433–440.
  • Dehais J, Shevchik S, Diem P, Mougiakakou SG. 2013. Food volume computation for self dietary assessment applications. IEEE 13th International Conference on Bioinformatics and Bioengineering (BIBE 2013); 2013 Nov 10–13; Chania, Greece. IEEE Computer Society; p. 1–4.
  • Dong Y. 2012. Tracking wrist motion to detect and measure the eating intake of free-living humans. Clemson: Clemson University.
  • Draelos Z, Qiu Q, Bronstein A, Sapiro G. 2015. Intel realsense = real low cost gaze. IEEE International Conference on Image Processing (ICIP 2015); 2015 Sept 27–30; Quebec, Canada. IEEE Computer Society; p. 2520–2524.
  • Eskin Y, Mihailidis A. 2012. An intelligent nutritional assessment system. 2012 AAAI Fall Symposium Series; 2012 Nov 2–4; Arlington, VA. AAAI; p. 2–7.
  • Fang S, Liu C, Zhu F, Boushey C, Delp E. 2015. A printer indexing system for color calibration with applications in dietary assessment. New trends in image analysis and processing-ICIAP 2015 Workshops. Amsterdam: Springer; p. 358–365.
  • Farinella GM, Allegra D, Stanco F. 2014. A benchmark dataset to study the representation of food images. Computer Vision-ECCV 2014 Workshops; 2014 Sept 6–7; Zurich, Switzerland. Switzerland: Springer; Vol. 8927, p. 584–599.
  • Farinella GM, Moltisanti M, Battiato S. 2014. Classifying food images represented as Bag of Textons. IEEE International Conference on Image Processing (ICIP 2014); 2014 Oct 27–30; Paris, France. IEEE Computer Society; p. 5212–5216.
  • Farinella GM, Moltisanti M, Battiato S. 2015. Food recognition using consensus vocabularies. New trends in image analysis and processing-ICIAP 2015 Workshops. Amsterdam: Springer; p. 384–392.
  • Farooq M, Chandler-Laney P, Hernandez-Reif M, Sazonov E. 2015. Monitoring of infant feeding behavior using a jaw motion sensor. J Healthc Eng. 6:23–40.
  • Farooq M, Fontana JM, Sazonov E. 2014. A novel approach for food intake detection using electroglottography. Physiol Meas. 35:739.
  • Farooq M, Sazonov E. 2016. Automatic measurement of chew count and chewing rate during food intake. Electronics. 5:62.
  • Fontana JM, Farooq M, Sazonov E. 2014. Automatic ingestion monitor: a novel wearable device for monitoring of ingestive behavior. IEEE Trans Biomed Eng. 61:1772–1779.
  • Food scanner prize – European Commission. 2015. Available from http://ec.europa.eu/research/horizonprize/index.cfm?prize=food-scanner
  • Ford AL, Bergh C, Södersten P, Sabin MA, Hollinghurst S, Hunt LP, Shield JP. 2010. Treatment of childhood obesity by retraining eating behaviour: randomised controlled trial. Brit Med J. 340:b5388
  • Frontczak M, Wargocki P. 2011. Literature survey on how different factors influence human comfort in indoor environments. Build Environ. 46:922–937.
  • Gletsu-Miller N, McCrory MA. 2014. Modifying eating behavior: novel approaches for reducing body weight, preventing weight regain, and reducing chronic disease risk. Adv Nutr. 5:789–791.
  • Hassannejad H, Matrella G, Ciampolini P,D, Munari I, Mordonini M, Cagnoni S. 2016. Food image recognition using very deep convolutional networks. ACM Multimedia Workshops - MADiMa 2016. 2016 Oct 16; Amsterdam, the Netherlands. New York (NY): ACM; p. 41–49.
  • Hassannejad H, Matrella G, Mordonini M, Cagnoni S. 2015a. A mobile app for food detection: new approach to interactive segmentation. Italy: Italian Forum of Ambient Assisted Living (ForItAAL); p. 306–313.
  • Hassannejad H, Matrella G, Mordonini M, Cagnoni S. 2015b. A stochastic approach to detect small checkerboards. AI* IA 2015 Advances in Artificial Intelligence: XIVth International Conference of the Italian Association for Artificial Intelligence; 2015 Sept 23–25; Ferrara, Italy. Switzerland: Springer; p. 75–86.
  • Hassannejad H, Matrella G, Mordonini M, Cagnoni S. 2015c. Using small checkerboards as size reference: a model-based approach. New trends in image analysis and processing - ICIAP 2015 Workshops - MADiMa 2015. Amsterdam: Springer; p. 393–400.
  • He H, Kong F, Tan J. 2016. DietCam: multiview food recognition using a MultiKernel SVM. IEEE J Biomed Health Inform. 20:848–855.
  • He Y, Xu C, Khanna N, Boushey CJ, Delp EJ. 2013. Context based food image analysis. 20th IEEE International Conference on Image Processing (ICIP 2013); 2013 Sept 15–18; Melbourne, Australia. IEEE Computer Society; p. 2748–2752.
  • Hoashi H, Joutou T, Yanai K. 2010. Image recognition of 85 food categories by feature fusion. IEEE International Symposium on Multimedia (ISM 2010); 2010 Dec 13–15; Taiwan. IEEE Computer Society; p. 296–301.
  • Hopcroft J, Tarjan R. 1973. Algorithm 447: efficient algorithms for graph manipulation. Commun ACM. 16:372–378.
  • Jarrett K, Kavukcuoglu K, Lecun Y. 2009. What is the best multi-stage architecture for object recognition? 2009 IEEE 12th International Conference on Computer Vision; 2009 Sept 27–Oct 4; Kyoto, Japan. IEEE Computer Society; p. 2146–2153.
  • Jia W, Yue Y, Fernstrom JD, Zhang Z, Yang Y, Sun M. 2012. 3D localization of circular feature in 2D image and application to food volume estimation. International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2012); 2012 Aug 28–Sep 1; San Diego, CA, USA. IEEE Computer Society; p. 4545–4548.
  • Joutou T, Yanai K. 2009. A food image recognition system with multiple kernel learning. 2009 16th IEEE International Conference on Image Processing (ICIP); 2009 Nov 7–10; Cairo, Egypt. IEEE Computer Society; p. 285–288.
  • Jovanov E, Sazonov E, Poon C. 2014. Sensors and systems for obesity care and research. 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014); 2014 Aug 26–30; Chicago, IL, USA. IEEE Computer Society; p. 3188–3191.
  • Kagaya H, Aizawa K. 2015. Highly accurate food/non-food image classification based on a deep convolutional neural network. New trends in image analysis and processing-ICIAP 2015 Workshops. Amsterdam: Springer; p. 350–357.
  • Kawano Y, Yanai K. 2014a. Automatic expansion of a food image dataset leveraging existing categories with domain adaptation. Computer Vision-ECCV 2014 Workshops; 2014 Sept 6–7; Zurich, Switzerland. Switzerland: Springer; p. 3–17.
  • Kawano Y, Yanai K. 2014b. Food image recognition with deep convolutional features. 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication; 2014 Sept 13–17; Seattle, WA, USA. p. 589–593.
  • Kawano Y, Yanai K. 2014c. FoodCam-256: a large-scale real-time mobile Food recognition system employing high-dimensional features and compression of classifier weights. ACM International Conference on Multimedia; 2014 Nov 3–7; Orlando, FL, USA. New York (NY): ACM; p. 761–762.
  • Kawano Y, Yanai K. 2014d. FoodCam: a real-time mobile food recognition system employing fisher vector. MultiMedia Modeling. Springer: Switzerland; p. 369–373.
  • Kitamura K, Yamasaki T, Aizawa K. 2009. FoodLog: capture, analysis and retrieval of personal food images via web. ACM Multimedia 2009 Workshop on Multimedia for Cooking and Eating Activities; 2009 Oct 19–24; Beijing, China. New York (NY): ACM. p. 23–30.
  • Kong F, Tan J. 2012. DietCam: automatic dietary assessment with mobile camera phones. Pervas Mob Comput. 8:147–163.
  • Krizhevsky A, Sutskever I, Hinton GE. 2012. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25: 26th Annual Conference on NIPS 2012; 2012 Dec 3–6; Nevada, USA. Red Hook, NY: Curran & Associates, Inc.
  • LeCun Y, Bengio Y, Hinton G. 2015. Deep learning. Nature. 521:436–444.
  • Li J, Zhang N, Hu L, Li Z, Li R, Li C, Wang S. 2011. Improvement in chewing activity reduces energy intake in one meal and modulates plasma gut hormone concentrations in obese and lean young Chinese men. Am J Clin Nutr. 94:709–716.
  • Liu J, Johns E, Atallah L, Pettitt C, Lo B, Frost G, Yang GZ. 2012. An intelligent food-intake monitoring system using wearable sensors. 2012 Ninth International Conference on Wearable and Implantable Body Sensor Networks; 2012 May 9–12; London, UK. p. 154–160.
  • Liu R. 2016. Food recognition and detection with minimum supervision. Canada: The University of Western Ontario.
  • Livingstone MBE, Black AE. 2003. Markers of the validity of reported energy intake. J Nutr. 133:895S–920S.
  • Lowe DG. 2004. Distinctive image features from scale-invariant keypoints. Int J Comp Vis. 60:91–110.
  • Makeyev O, Lopez-Meyer P, Schuckers S, Besio W, Sazonov E. 2012. Automatic food intake detection based on swallowing sounds. Biomed Signal Process Control. 7:649–656.
  • Maramis C, Diou C, Ioakeimidis I, Lekka I, Dudnik G, Mars M, Maglaveras N, Bergh C, Delopoulos A. 2014. Preventing obesity and eating disorders through behavioural modifications: The SPLENDID vision. EAI 4th International Conference on Wireless Mobile Communication and Healthcare (Mobihealth 2014); 2014 Nov 3–5; Athens, Greece. p. 7–10.
  • Martin CK, Kaya S, Gunturk BK. 2009. Quantification of food intake using food image analysis. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2009 Sept 3–6; Minneapolis, MN USA. IEEE Computer Society.
  • Matsuda Y, Yanai K. 2012. Multiple-food recognition considering co-occurrence employing manifold ranking. 21st International Conference on Pattern Recognition (ICPR 2012); 2012 Nov 11–15; Tsukuba, Japan. p. 2017–2020.
  • Mesas A, Muñoz-Pareja M, López-García E, Rodríguez-Artalejo F. 2012. Selected eating behaviours and excess body weight: a systematic review. Obes Rev. 13:106–135.
  • Movassaghi S, Abolhasan M, Lipman J, Smith D, Jamalipour A. 2014. Wireless body area networks: a survey. IEEE Commun Surv Tut. 16:1658–1686.
  • Muthukrishnan R, Radha M. 2011. Edge detection techniques for image segmentation. Int J Comp Sci Inf Techol. 3:259.
  • Nadeem A, Hussain MA, Owais O, Salam A, Iqbal S, Ahsan K. 2015. Application specific study, analysis and classification of body area wireless sensor network applications. Comp Networks. 83:363–380.
  • Nederkoorn C, Smulders F, Jansen A. 1999. Recording of swallowing events using electromyography as a non-invasive measurement of salivation. Appetite. 33:361–369.
  • Nutritional Epidemiology Group. 2015. Making the best use of new technologies in the National Diet and Nutrition Survey: a review. Leeds (UK): University of Leeds.
  • O'Hara S, Draper BA. 2011. Introduction to the bag of features paradigm for image classification and retrieval. arXiv Preprint arXiv. 1101.3354.
  • Okamoto K, Yanai K. 2016. GrillCam: a real-time eating action recognition system. International Conference on Multimedia Modeling; 2016 Jan 4–6; Miami, FL, USA. p. 331–335.
  • Oliveira L, Costa V, Neves G, Oliveira T, Jorge E, Lizarraga M. 2014. A mobile, lightweight, poll-based food identification system. Pattern Recognit. 47:1941–1952.
  • Papapanagiotou V, Diou C, Lingchuan Z, van den Boer J, Mars M, Delopoulos A. 2015. Fractal nature of chewing sounds. New Trends in Image Analysis and Processing-ICIAP 2015 Workshops. Amsterdam: Springer; p. 401–408.
  • Päßler S, Fischer WJ. 2011. Food intake activity detection using a wearable microphone system. 7th International Conference on Intelligent Environments (IE 2011); 2011 July 25–28; Nottingham, UK. p. 298–301.
  • Päßler S, Fischer WJ. 2014. Food intake monitoring: automated chew event detection in chewing sounds. IEEE J Biomed Health Inform. 18:278–289.
  • Pettitt C, Liu J, Kwasnicki RM, Yang G-Z, Preston T, Frost G. 2016. A pilot study to determine whether using a lightweight, wearable micro-camera improves dietary assessment accuracy and offers information on macronutrients and eating rate. Brit J Nutr. 115:160–167.
  • Pouladzadeh P, Yassine A, Shirmohammadi S. 2015. FooDD: Food detection dataset for calorie measurement using food images. New trends in image analysis and processing-ICIAP 2015 Workshops. Amsterdam: Springer; p. 441–448.
  • Puri M, Zhiwei Z, Yu Q, Divakaran A, Sawhney H. 2009. Recognition and volume estimation of food intake using a mobile device. Proceedings of the Workshop on Applications of Computer Vision (WACV); 2009 Dec 7–8; Snowbird, UT, USA. IEEE Computer Society.
  • Rahman MH, Li Q, Pickering M, M, F Kerr D, Bouchey C, Delp E. 2012. Food volume estimation in a mobile phone based dietary assessment system. Proceedings of the Eighth International Conference on Signal Image Technology and Internet Based Systems; 2012 Nov 25–29; Naples, Italy. IEEE Computer Society.
  • Rahmana MH, Pickering MR, Kerr D, Boushey CJ, Delp EJ. 2012. A new texture feature for improved food recognition accuracy in a mobile phone based dietary assessment system. IEEE International Conference on Multimedia and Expo Workshops (ICMEW 2012); 2012 July 9–13; Melbourne, Australia. IEEE Computer Society; p. 418–423.
  • Ramos-Garcia RI, Hoover AW. 2013. A study of temporal action sequencing during consumption of a meal. International Conference on Bioinformatics, Computational Biology and Biomedical Informatics; 2013 Sept 22–25; Washington DC, USA. New York (NY): ACM. p. 68:68–68.
  • Rhyner D, Loher H, Dehais J, Anthimopoulos M, Shevchik S, Botwey RH, Duke D, et al. 2016. Carbohydrate estimation by a mobile phone-based system versus self-estimations of individuals with type 1 diabetes mellitus: a comparative study. J Med Internet Res. 18:e101.
  • Robinson E, Almiron-Roig E, Rutters F, de Graaf C, Forde CG, Smith CT, Nolan SJ, Jebb SA. 2014. A systematic review and meta-analysis examining the effect of eating rate on energy intake and hunger. Am J Clin Nutr. 100:123–151.
  • Santos JL, Ho-Urriola JA, González A, Smalley SV, Domínguez-Vásquez P, Cataldo R, Obregón AM, et al. 2011. Association between eating behavior scores and obesity in Chilean children. Nutr J. 10:10:1186.
  • Sazonov E, Neuman MR. 2014. Wearable sensors: fundamentals, implementation and applications. Amsterdam: Elsevier.
  • Sazonov ES, Fontana JM. 2012. A sensor system for automatic detection of food intake through non-invasive monitoring of chewing. IEEE Sens J. 12:1340–1348.
  • Sazonov ES, Schuckers SA, Lopez-Meyer P, Makeyev O, Melanson EL, Neuman MR, Hill JO. 2009. Toward objective monitoring of ingestive behavior in free-living population. Obesity (Silver Spring). 17:1971–1975.
  • Schoeller DA. 1995. Limitations in the assessment of dietary energy intake by self-report. Metab Clin Exp. 44:18–22.
  • Sharma S, Jasper P, Muth E, Hoover A. 2016. Automatic detection of periods of eating using wrist motion tracking. IEEE First International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE 2016); 2016 June 27–19; Washington DC, USA. IEEE Computer Society; p. 362–363.
  • Shroff G, Smailagic A, Siewiorek DP. 2008. Wearable context-aware food recognition for calorie monitoring. 2008 12th IEEE International Symposium on Wearable Computers; 2008 Sept 28–Oct 1; Pittsburgh, PA, USA. IEEE Computer Society; p. 119–120.
  • Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. 2015. Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition; 2015 June 7–12; Bosto, MA, USA. IEEE Computer Society; p. 1–9.
  • Villalobos G, Almaghrabi R, Hariri B, Shirmohammadi S. 2011. A personal assistive system for nutrient intake monitoring. 2011 International ACM Workshop on Ubiquitous Meta User Interfaces; 2011 Nov 28–Dec 1; Scottsdale, AZ, USA. p. 17–22.
  • Wang Y, He Y, Zhu F, Boushey C, Delp E. 2015. The use of temporal information in food image analysis. New trends in image analysis and processing-ICIAP 2015 Workshops. Amsterdam: Springer; p. 317–325.
  • Wazumi M, Han X-H, Ai D, Chen Y-W. 2011. Auto-recognition of food images using SPIN feature for Food-Log system. 6th International Conference on Computer Sciences and Convergence Information Technology (ICCIT 2011); 2011 Nov 29–Dec 1; Seogwipo, Korea. p. 874–877.
  • World Health Organization. 2016. Available from http://www.who.int/mediacentre/factsheets/fs312/en/
  • Witschi JC. 1990. Short-term dietary recall and recording methods. Nutrition Epidemiol. 4:52–68.
  • Wu W, Yang J. 2009. Fast food recognition from videos of eating for calorie estimation. 2009 IEEE International Conference on Multimedia and Expo; 2009 June 28–July 3; New York, NY, USA. p. 1210–1213.
  • Xu C, He Y, Khannan N, Parra A, Boushey C, Delp E. 2013. Image-based food volume estimation. 5th International Workshop on Multimedia for Cooking & Eating Activities; 2013 Oct 21–25; Barcelona, Spain. p. 75–80.
  • Yanai K, Kawano Y. 2015. Food image recognition using deep convolutional network with pre-training and fine-tuning. IEEE International Conference on Multimedia & Expo Workshops (ICME 2015); 2015 June 29–July 3; Torino, Italy. IEEE Computer Society; p. 1–6.
  • Yao N. 2011. Food dimension estimation from a single image using structured lights. Pittsburgh, PA: University of Pittsburgh.
  • Yogamangalam BKR, Karthikeyan B. 2013. Segmentation technique comparison in image processing. Int J Eng Technol. 5:307–313.
  • Zhang MM. 2011. Identifying the cuisine of a plate of food. Technical Report CSE. 190. La Jolla (CA): University of California at San Diego.
  • Zhang R, Freund M, Amft O, Cheng J, Zhou B, Lukowicz P, Fernando S, Chabrecek P. 2016. A generic sensor fabric for multi-modal swallowing sensing in regular upper-body shirts. Proceedings of the 2016 ACM International Symposium on Wearable Computers; 2016 Sept 12–16; Heidelberg, Germany. New York (NY): ACM; p. 46–47.
  • Zhu F, Bosch M, Khanna N, Boushey CJ, Delp EJ. 2015. Multiple hypotheses image segmentation and classification with application to dietary assessment. IEEE J Biomed Health Inform. 19:377–388.
  • Zhu F, Bosch M, Schap T, Khanna N, Ebert DS, Boushey CJ, Delp EJ. 2011. Segmentation assisted food classification for dietary assessment. Proc SPIE Int Soc Opt Eng. 7873:78730B.
  • Zhu F, Bosch M, Woo I, Kim S, Boushey CJ, Ebert DS, Delp EJ. 2010. The use of mobile devices in aiding dietary assessment and evaluation. IEEE J Sel Top Signal Process. 4:756–766.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.