Uploaded March 2024 | Updated September 2026, 4 hours ago
Microsoft Future Leaders in Robotics and AI Seminar Series: Structured World Models for Robotics
Online Seminar
Krishna Murthy Jatavallabhula
Postdoctoral Fellow
Massachusetts Institute of Technology
Humans have an innate ability to construct detailed mental representations of the world from limited sensory data. These 'world models' are central to natural intelligence, allowing us to perceive, reason about, and act in the physical world. My research seeks to create 'computational world models' -- artificial intelligence techniques that enable robots to understand and operate in the world around as effectively as humans. Despite the impressive successes of modern machine learning approaches on domains such as text, images, and video---where abundant training data is readily available---these have not translated to robotics. Building generally capable robotic systems presents unique challenges including the lack of data, and the need to adapt learning algorithms to a wide variety of embodiments, environments, and tasks of interest.
In my talk, I will present how my research contributes to the design of computational models for spatial, physical, and multimodal understanding. I will discuss differentiable computing approaches that have advanced the field of spatial perception, enabling an understanding of the structure of the 3D world, its constituent objects, and their semantic and physical properties from videos. I will also detail how my work interfaces advances in large image, language, and audio models with 3D scenes, enabling robots and computer vision systems to flexibly query these structured world models for a wide range of tasks. Finally, I will outline my vision for the future, where structured world models and modern scaling-based approaches work in tandem to create versatile robot perception and planning algorithms with the potential to meet and ultimately surpass human-level capabilities.
For more information on the Maryland Robotics Center see:
https://robotics.umd.edu
Microsoft Future Leaders in Robotics and AI Seminar Series: Structured World Models for Robotics
Online Seminar
Krishna Murthy Jatavallabhula
Postdoctoral Fellow
Massachusetts Institute of Technology
Humans have an innate ability to construct detailed mental representations of the world from limited sensory data. These 'world models' are central to natural intelligence, allowing us to perceive, reason about, and act in the physical world. My research seeks to create 'computational world models' -- artificial intelligence techniques that enable robots to understand and operate in the world around as effectively as humans. Despite the impressive successes of modern machine learning approaches on domains such as text, images, and video---where abundant training data is readily available---these have not translated to robotics. Building generally capable robotic systems presents unique challenges including the lack of data, and the need to adapt learning algorithms to a wide variety of embodiments, environments, and tasks of interest.
In my talk, I will present how my research contributes to the design of computational models for spatial, physical, and multimodal understanding. I will discuss differentiable computing approaches that have advanced the field of spatial perception, enabling an understanding of the structure of the 3D world, its constituent objects, and their semantic and physical properties from videos. I will also detail how my work interfaces advances in large image, language, and audio models with 3D scenes, enabling robots and computer vision systems to flexibly query these structured world models for a wide range of tasks. Finally, I will outline my vision for the future, where structured world models and modern scaling-based approaches work in tandem to create versatile robot perception and planning algorithms with the potential to meet and ultimately surpass human-level capabilities.
For more information on the Maryland Robotics Center see:
https://robotics.umd.edu
