Uploaded April 2024 | Updated September 2026, 3 hours ago
Microsoft Future Leaders in Robotics and AI Seminar Series: Sense, Imagine, Act: Multimodal Perception & Reinforcement Learning for UxV Guidance & Control
Online Seminar
Elena Shrestha
Postdoctoral Fellow
University of Michigan
While reinforcement learning offers potential for continual learning and adaptability in complex scenarios, its application to real-world robotics faces significant challenges. Unlike in simulations, physical platforms struggle to collect a diverse corpus of training data due to critical safety risks and the inherent constraints of operating within a dynamic and partially observable environment. My work draws inspiration from the human capability to fuse and exploit multiple sensing modalities, construct comprehensive models of how the world operates, and then leverage those models to adeptly navigate in challenging and often unpredictable environments. In this seminar, I will present an overview of how unmanned vehicles (ground, air, and surface) can exploit a world model constructed through multimodal perception, to learn near-optimal policies for guidance and control. A key aspect of the approach is learning from imagination in which the world model is used to simulate future imagined trajectories, enabling it to anticipate potential risks before encountering them in the real world. My ongoing work and long-term vision is to evolve the traditional sense-plan-act framework into a more intuitive and cognitively inspired sense-imagine-act model.
For more information on the Maryland Robotics Center see:
https://robotics.umd.edu
Microsoft Future Leaders in Robotics and AI Seminar Series: Sense, Imagine, Act: Multimodal Perception & Reinforcement Learning for UxV Guidance & Control
Online Seminar
Elena Shrestha
Postdoctoral Fellow
University of Michigan
While reinforcement learning offers potential for continual learning and adaptability in complex scenarios, its application to real-world robotics faces significant challenges. Unlike in simulations, physical platforms struggle to collect a diverse corpus of training data due to critical safety risks and the inherent constraints of operating within a dynamic and partially observable environment. My work draws inspiration from the human capability to fuse and exploit multiple sensing modalities, construct comprehensive models of how the world operates, and then leverage those models to adeptly navigate in challenging and often unpredictable environments. In this seminar, I will present an overview of how unmanned vehicles (ground, air, and surface) can exploit a world model constructed through multimodal perception, to learn near-optimal policies for guidance and control. A key aspect of the approach is learning from imagination in which the world model is used to simulate future imagined trajectories, enabling it to anticipate potential risks before encountering them in the real world. My ongoing work and long-term vision is to evolve the traditional sense-plan-act framework into a more intuitive and cognitively inspired sense-imagine-act model.
For more information on the Maryland Robotics Center see:
https://robotics.umd.edu
