Uploaded January 2024 | Updated September 2026, 3 hours ago
Maryland Robotics Center Seminar: Robot Foundation Models: Training Robots with Internet Scale Data
Jie Tan
Senior Staff Research Scientist
Google
Foundation Models, after training with Internet-scale data, show excellent understanding of natural language and images, possessing common sense, and performing logical reasoning and predictions. All of these capabilities are essential for developing intelligent and autonomous robots. How can we tap into the power of these Foundation Models in robotics? In this talk, I will cover three recent papers from Google DeepMind about building Robot Foundation Models: SayCan, RT-2, and ROSIE. They not only demonstrate how to ground large language models (LLM), and vision-language models (VLM) based on the robots' physical capability and the real-world environments, but also discuss how to leverage AI generative models for large-scale data augmentation. Pre-trained with Internet-scale text and image data, and fine-tuned with robotic data collected in the real world or imagined through diffusion models, the Robot Foundation Models show unprecedented capabilities for long-horizon task planning, and generalizable low-level skills.
For more information on the Maryland Robotics Center see:
https://robotics.umd.edu
Maryland Robotics Center Seminar: Robot Foundation Models: Training Robots with Internet Scale Data
Jie Tan
Senior Staff Research Scientist
Foundation Models, after training with Internet-scale data, show excellent understanding of natural language and images, possessing common sense, and performing logical reasoning and predictions. All of these capabilities are essential for developing intelligent and autonomous robots. How can we tap into the power of these Foundation Models in robotics? In this talk, I will cover three recent papers from Google DeepMind about building Robot Foundation Models: SayCan, RT-2, and ROSIE. They not only demonstrate how to ground large language models (LLM), and vision-language models (VLM) based on the robots' physical capability and the real-world environments, but also discuss how to leverage AI generative models for large-scale data augmentation. Pre-trained with Internet-scale text and image data, and fine-tuned with robotic data collected in the real world or imagined through diffusion models, the Robot Foundation Models show unprecedented capabilities for long-horizon task planning, and generalizable low-level skills.
For more information on the Maryland Robotics Center see:
https://robotics.umd.edu
