Uploaded March 2022 | Updated September 2026, 15 hours ago
Recent advances in vision and language modeling have been powered by truly massive datasets, often mined from the web. Instruction-following robots will also require large amounts of data to train and evaluate. However, images/videos/documents found on the web do not satisfy the needs of embodied agents, and data annotation is slow and expensive. Synthetic data is a promising alternative. In this talk, I’ll discuss two specific approaches. First, I’ll introduce our work generating synthetic navigation instructions at near-human quality, which can be used to train instruction-following navigation agents at large-scale. Second, I’ll show how novel view synthesis can be used to create new trajectories for visual navigation agents, without requiring a simulation environment. Together, these approaches promise to greatly expand the available high-quality training data for language-guided agents.
Recent advances in vision and language modeling have been powered by truly massive datasets, often mined from the web. Instruction-following robots will also require large amounts of data to train and evaluate. However, images/videos/documents found on the web do not satisfy the needs of embodied agents, and data annotation is slow and expensive. Synthetic data is a promising alternative. In this talk, I’ll discuss two specific approaches. First, I’ll introduce our work generating synthetic navigation instructions at near-human quality, which can be used to train instruction-following navigation agents at large-scale. Second, I’ll show how novel view synthesis can be used to create new trajectories for visual navigation agents, without requiring a simulation environment. Together, these approaches promise to greatly expand the available high-quality training data for language-guided agents.
