Future Interfaces Group | Enhancing Mobile Voice Assistants with WorldGaze @FiglabCMU | Uploaded 4 years ago | Updated 2 hours ago
Contemporary voice assistants require that objects of interest be specified in spoken commands. Of course, users are often looking directly at the object or place of interest – fine-grained, contextual information that is currently unused. We present WorldGaze, a software-only method for smartphones that provides the real-world gaze location of a user that voice agents can utilize for rapid, natural, and precise interactions. We achieve this by simultaneously opening the front and rear cameras of a smartphone. The front-facing camera is used to track the head in 3D, including estimating its direction vector. As the geometry of the front and back cameras are fixed and known, we can raycast the head vector into the 3D world scene as captured by the rear-facing camera. This allows the user to intuitively define an object or region of interest using their head gaze. We started our investigations with a qualitative exploration of competing methods, before developing a functional, real-time implementation. We conclude with an evaluation that shows WorldGaze can be quick and accurate, opening new multimodal gaze+voice interactions for mobile voice agents.
Download Paper PDF: http://sven-mayer.com/wp-content/uploads/2020/03/mayer2020worldgaze.pdf
Mayer, Sven; Laput, Gierad; Harrison, Chris. 2020. Enhancing Mobile Voice Assistants with WorldGaze. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20), ACM, New York, NY, USA.
Contemporary voice assistants require that objects of interest be specified in spoken commands. Of course, users are often looking directly at the object or place of interest – fine-grained, contextual information that is currently unused. We present WorldGaze, a software-only method for smartphones that provides the real-world gaze location of a user that voice agents can utilize for rapid, natural, and precise interactions. We achieve this by simultaneously opening the front and rear cameras of a smartphone. The front-facing camera is used to track the head in 3D, including estimating its direction vector. As the geometry of the front and back cameras are fixed and known, we can raycast the head vector into the 3D world scene as captured by the rear-facing camera. This allows the user to intuitively define an object or region of interest using their head gaze. We started our investigations with a qualitative exploration of competing methods, before developing a functional, real-time implementation. We conclude with an evaluation that shows WorldGaze can be quick and accurate, opening new multimodal gaze+voice interactions for mobile voice agents.
Download Paper PDF: http://sven-mayer.com/wp-content/uploads/2020/03/mayer2020worldgaze.pdf
Mayer, Sven; Laput, Gierad; Harrison, Chris. 2020. Enhancing Mobile Voice Assistants with WorldGaze. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20), ACM, New York, NY, USA.
