Building Foundation Models for Vision and Robotics – Aniruddha Kembhavi @HopkinsEngineer
Building Foundation Models for Vision and Robotics – Aniruddha Kembhavi  @HopkinsEngineer
Uploaded August 2024 | Updated November 2026, 3 hours ago
Gerald M. Masson Distinguished Lecture Series
March 26, 2024

“Building Foundation Models for Vision and Robotics”
Aniruddha Kembhavi, Allen Institute for AI

Large language models support a whole gamut of tasks in natural language. In contrast, unification has been more challenging in computer vision, partly due to the heterogeneity of tasks in the visual domain and a scarcity of robotics data in the physical world. How do we create powerful unified systems for vision and robotics that can be as capable and creative as their language counterparts? Aniruddha Kembhavi will first present his work on Unified IO, the first single neural model to perform a large and diverse set of AI tasks spanning classical computer vision, image synthesis, vision-and-language, and natural language processing, and his follow-up work, Unified IO 2, that brings in video, audio, and action. Then he will present his recent surprising finding in robotics: Imitating shortest path planners in simulation can produce agents that can proficiently navigate, explore, and manipulate objects in the real world, with no human data or RL and with just RGB sensing—made possible by his recent works on ProcTHOR, HoloDeck, and ObjaVerse. Finally, he will present a compelling alternative to building unified models, Visual Programming, which uses language models as code generators to invoke smaller specialized vision models. This paradigm is efficient and effective, leverages models sourced from the entire community, and scales easily to large sets of diverse tasks.

Aniruddha Kembhavi is the senior director of computer vision at the Allen Institute for AI in Seattle. He is also an affiliate associate professor in the Department of Computer Science and Engineering at the University of Washington. He is interested in research problems at the intersection of vision, language, and embodiment. He received his PhD from the University of Maryland under the supervision of Professor Larry Davis and spent several years at Microsoft Bing building large-scale machine learning systems for image and video search. Kembhavi’s work has won several awards, including a Best Paper Award at the 2023 Conference on Computer Vision and Pattern Recognition and an Outstanding Paper Award at the 2022 Conference and Workshop on Neural Information Processing Systems.
Building Foundation Models for Vision and Robotics – Aniruddha KembhaviTowards Trustworthy AI: Provably Robust Extrapolation for Decision-Making – Anqi “Angie” LiuRecycling Fine-Tuned Models to Pretrain – Leshem ChoshenDesigning Machine Learning Processes for Equitable Health Systems – Marzyeh GhassemiBuilding Haystacksto Find Needles – Erik RyeMechanical Engineering Freshman Design Exhibition 2026Good Research Karma: The Unexpected Benefits of Striving for Algorithmic Fairness – Omer ReingoldCreating Robustness and Resiliency in Distributed Autonomous Groups of Robots or DronesAdversarial Robustness and Forensics for Deep Neural Networks – Ben Y. ZhaoMachine Learning & Causality: Building Efficient, Reliable Models for Decision-Making – Maggie MakarThe 2026 Charles and Mary O’Melia LectureData-Distributional Approaches for Generalizable Language Models – Sang Michael Xie
Johns Hopkins Engineering |

Building Foundation Models for Vision and Robotics – Aniruddha Kembhavi

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER