Toward Evaluating and Securing XR Experiences in Real Time with Large Multimodal Models (LMMs) @AWEXR
Toward Evaluating and Securing XR Experiences in Real Time with Large Multimodal Models (LMMs)  @AWEXR
Uploaded July 2026 | Updated November 2026, 12 hours ago
When deployed in unconstrained real-world environments, modern mixed reality (MR) experiences can fail in subtle but consequential ways: virtual objects may drift or occlude critical real-world elements; spatial audio cues may contradict visual content; MR-based guidance may suggest actions that are impossible, unsafe, or societally unacceptable. Such highly context-dependent failures, where MR content that is appropriate in one setting becomes unsafe in another, cannot be detected by classical signal processing or conventional ML approaches alone. Recent large multimodal models (LMMs) enable semantic, context-aware monitoring of immersive systems, going far beyond what was previously feasible. By jointly reasoning over video streams captured by MR devices, rendered virtual content, spatial audio, and scene context, these models can identify misaligned overlays, unsafe occlusions, inconsistent audio-visual cues, and policy violations in running MR applications.

In this talk, we present an LMM-based MR experience evaluation pipeline that combines foundation-scale multimodal models with auxiliary ML components to analyze continuous multimodal streams and trigger corrective actions when problematic content is detected. A central technical challenge in LMM-based MR evaluation is achieving sufficiently low latency for integration into live MR systems. We describe a system architecture and inference strategy that enables responsive, near–real-time multimodal monitoring while maintaining the breadth of semantic reasoning provided by large models.

The methodology presented in this talk received the 2025 IEEE International Symposium on Mixed and Augmented Reality (IEEE ISMAR) Best Paper Award. This research is funded by an NSF AI Institute, a DARPA Young Faculty Award, and a DARPA Director’s Fellowship.

Session Speaker:
- Maria Gorlatova, James N. & Elizabeth H. Barton Associate Professor of Electrical & Computer Engineering, Duke University

This session was recorded at AWE USA 2026 - the world's leading XR + AI event series. To learn more visit: awexr.com
Toward Evaluating and Securing XR Experiences in Real Time with Large Multimodal Models (LMMs)What we learned from LBE XR - Current and Future by Kevin Williams from KWPBuilding a Global Ecosystem for Physical AI by Ryan Wang and Michael GoldGoogle Keynote AWE USA 2026: Unifying the Android XR Ecosystem with Juston Payne & Hugo SwartInside the Immersive Video Renaissance: Tools, Stories and AudiencesXR Women Awards at AWE USA 2026Deploying XR Devices with PatientsAWE Nite Chicago: Immersive Technology of TravelThe XR-Robot Venn Diagram by Joe Michaels from 1HMXThe Missing Link in Smart Glasses by Paul Meissner from Applied MaterialsAWE USA 2026 Expo Hall Media TourXR, AI and the Next Consumer Journey
AWE XR |

Toward Evaluating and Securing XR Experiences in Real Time with Large Multimodal Models (LMMs)

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER