Gaze-Informed location-based services

ABSTRACT

Location-based services (LBS) provide more useful, intelligent assistance to users by adapting to their geographic context. For some services that context goes beyond a location and includes further spatial parameters, such as the user’s orientation or field of view. Here, we introduce Gaze-Informed LBS (GAIN-LBS), a novel type of LBS that takes into account the user’s viewing direction. Such a system could, for instance, provide audio information about the specific building a tourist is looking at from a vantage point. To determine the viewing direction relative to the environment, we record the gaze direction relative to the user’s head with a mobile eye tracker. Image data from the tracker’s forward-looking camera serve as input to determine the orientation of the head w.r.t. the surrounding scene, using computer vision methods that allow one to estimate the relative transformation between the camera and a known view of the scene in real-time and without the need for artificial markers or additional sensors. We focus on how to map the point of regard of a user to a reference system, for which the objects of interest are known in advance. In an experimental validation on three real city panoramas, we confirm that the approach can cope with head movements of varying speed, including fast rotations up to to 63 degrees per second. We further demonstrate the feasibility of GAIN-LBS for tourist assistance with a proof-of-concept experiment in which a tourist explores a city panorama, where the approach achieved a recall that reaches over 99%. Finally, a GAIN-LBS can provide objective and qualitative ways of examining the gaze of a user based on what the user is currently looking at.

KEYWORDS:

• Location-based services
• eye tracking
• point-of-regard estimation
• gaze-based interaction
• geographic human–computer interaction

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. https://www.microsoft.com/en-us/research/product/computational-photography-applications/image-composite-editor/.

2. To limit distortions that impair image matching, cylindrical panoramas are limited to in vertical direction, conventional wide-angle perspective images are limited both vertically and horizontally.

3. Other projections could also be used.

4. http://www.eyetracking-glasses.com.

5. http://opencv.org.

6. Recall is defined as the percentage of frames in which the user observes building A and the system correctly identified it as ‘AOI A’.

Additional information

Funding

This work was supported by the ETH Zurich Research Grant [ETH-38 14-2]; the Horizon 2020 Framework Programme [687757 - REPLICATE.].