Abstract
Corneal Surface Reflections, or reflections on our eye-surface, have been shown as a valid and more socially acceptable source of information for passive lifelogging applications by prior work. However, automatic analysis of corneal surface reflections from a single RGB camera to support passive lifelogging is not extensively investigated in prior work. To address this, we developed a synthetic and self-supervised learning-based two-stage pipeline of deep learning models to detect objects in these reflections. Our prototype only consists a single RGB camera looking into the eye. We collected data from different users in uncontrolled environments using the prototype and trained our system to detect multiple classes of objects present in a typical office environment. We then evaluated our model in partially-controlled and in-the-wild scenarios. In addition, based on the findings from a follow up user study and prior work, we discuss strengths and weaknesses of our system and using corneal surface reflections for passive lifelogging. Finally, we opensource our source codes and trained checkpoints.
Notes
Acknowledgments
We are thankful to all the supporters of this work including the study participants.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Notes
2 Access to the dataset is granted upon request to authors via [email protected]
Additional information
Funding
Notes on contributors
Tharindu Kaluarachchi
Tharindu Kaluarachchi completed his PhD at The University of Auckland. His research interests lie in developing AI applications for non-AI-experts using the Human-Centered Machine Learning approach. Currently Tharindu works as a Chief Technology Officer (CTO) in Singapore, developing AI solutions for the Tea Industry.
Shamane Siriwardhana
Shamane Siriwardhana completed his PhD at The University of Auckland. His research interests are self-supervised learning, multimodal deep learning, emotion recognition, unsupervised learning, natural language understanding, and human-computer interaction.
Elliott Wen
Elliott Wen joined the Augmented Human Lab in Auckland Bioengineering Institute at the University of Auckland as a Research Fellow. His research interests include Software Engineering, Pervasive Computing, and Mobile Sensing. Elliott obtained his PhD degree from the University of Auckland.
Suranga Nanayakkara
Suranga Nanayakkara is an Associate Professor at Department of Information Systems & Analytics, School of Computing at National University of Singapore. He founded the “Augmented Human Lab” to explore ways of designing intelligent human-computer interfaces that extend the limits of our perceptual and cognitive capabilities.