A Corneal Surface Reflections-Based Intelligent System for Lifelogging Applications

Abstract

Corneal Surface Reflections, or reflections on our eye-surface, have been shown as a valid and more socially acceptable source of information for passive lifelogging applications by prior work. However, automatic analysis of corneal surface reflections from a single RGB camera to support passive lifelogging is not extensively investigated in prior work. To address this, we developed a synthetic and self-supervised learning-based two-stage pipeline of deep learning models to detect objects in these reflections. Our prototype only consists a single RGB camera looking into the eye. We collected data from different users in uncontrolled environments using the prototype and trained our system to detect multiple classes of objects present in a typical office environment. We then evaluated our model in partially-controlled and in-the-wild scenarios. In addition, based on the findings from a follow up user study and prior work, we discuss strengths and weaknesses of our system and using corneal surface reflections for passive lifelogging. Finally, we opensource our source codes and trained checkpoints.

Notes

Acknowledgments

We are thankful to all the supporters of this work including the study participants.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 https://docs.pupil-labs.com//#diy

2 Access to the dataset is granted upon request to authors via [email protected]

3 https://drive.google.com/file/d/1jTOmjkd8jTEbrxf/_6sVbtA-VG5bsLlhv/view?usp=sharing, https://drive.google.com/file/d/11ugbXqMZQVC1SwoUHF1pzeNtzY4WM64y/view?usp= sharing

4 https://docs.pupil-labs.com//#diy

5 https://www.amazon.com/Microsoft-LifeCam-HD-6000-Webcam-Notebooks/dp/B00372567A

6 https://github.com/google-research/simclr

7 https://www.tobiipro.com/product-listing/tobii-pro-glasses-3/

8 https://design.google/library/ux-ai/

9 https://www.digikey.com/en/products/detail/omnivision-technologies-inc/OVM7692-RAAA/3190565

Additional information

Funding

This work was supported by the the Tertiary Education Commission, New Zealand under the Assistive Augmentations Grant 3716751.

Notes on contributors

Tharindu Kaluarachchi

Tharindu Kaluarachchi completed his PhD at The University of Auckland. His research interests lie in developing AI applications for non-AI-experts using the Human-Centered Machine Learning approach. Currently Tharindu works as a Chief Technology Officer (CTO) in Singapore, developing AI solutions for the Tea Industry.

Shamane Siriwardhana

Shamane Siriwardhana completed his PhD at The University of Auckland. His research interests are self-supervised learning, multimodal deep learning, emotion recognition, unsupervised learning, natural language understanding, and human-computer interaction.

Elliott Wen

Elliott Wen joined the Augmented Human Lab in Auckland Bioengineering Institute at the University of Auckland as a Research Fellow. His research interests include Software Engineering, Pervasive Computing, and Mobile Sensing. Elliott obtained his PhD degree from the University of Auckland.

Suranga Nanayakkara

Suranga Nanayakkara is an Associate Professor at Department of Information Systems & Analytics, School of Computing at National University of Singapore. He founded the “Augmented Human Lab” to explore ways of designing intelligent human-computer interfaces that extend the limits of our perceptual and cognitive capabilities.