Uploaded February 2025 | Updated September 2026, 3 days ago
Abstract: Imitation learning has driven the development of generalist policies capable of autonomously solving multiple tasks. However, when a pretrained policy makes errors during deployment, there are limited mechanisms for users to customize its behavior. While collecting additional data for fine-tuning can address such issues, doing so for each downstream use case is inefficient at scale. My research proposes an alternative perspective: framing policy errors as task mis-specifications rather than skill deficiencies. By
enabling users to specify tasks unambiguously at inference-time, the appropriate skill for a given context can be retrieved without fine-tuning. Specifically, I propose (1) inference-time steering, which leverages human interactions for single-step task specification, and (2) task and motion imitation, which uses symbolic plans for multi-step task specification. These frameworks correct misaligned policy predictions without requiring additional training, maximizing the utility of pretrained models while achieving inference-time user objectives.
Bio: Felix Yanwei Wang is a final-year PhD candidate in Electrical Engineering and Computer Science (EECS) at MIT, advised by Prof. Julie Shah. His research focuses on adapting pretrained manipulation policies for human-robot interaction. He earned his Bachelor's degree from Middlebury College and his Master's degree from Northwestern University. He has also worked under the guidance of Prof. Dieter Fox at the NVIDIA Robotics Lab. Felix is a recipient of the MIT Presidential Fellowship and the Work of the Future Fellowship in Generative AI at MIT. His research has been recognized with oral and spotlight presentations at CoRL and ICLR, featured on PBS, and is currently exhibited at the MIT Museum.
Abstract: Imitation learning has driven the development of generalist policies capable of autonomously solving multiple tasks. However, when a pretrained policy makes errors during deployment, there are limited mechanisms for users to customize its behavior. While collecting additional data for fine-tuning can address such issues, doing so for each downstream use case is inefficient at scale. My research proposes an alternative perspective: framing policy errors as task mis-specifications rather than skill deficiencies. By
enabling users to specify tasks unambiguously at inference-time, the appropriate skill for a given context can be retrieved without fine-tuning. Specifically, I propose (1) inference-time steering, which leverages human interactions for single-step task specification, and (2) task and motion imitation, which uses symbolic plans for multi-step task specification. These frameworks correct misaligned policy predictions without requiring additional training, maximizing the utility of pretrained models while achieving inference-time user objectives.
Bio: Felix Yanwei Wang is a final-year PhD candidate in Electrical Engineering and Computer Science (EECS) at MIT, advised by Prof. Julie Shah. His research focuses on adapting pretrained manipulation policies for human-robot interaction. He earned his Bachelor's degree from Middlebury College and his Master's degree from Northwestern University. He has also worked under the guidance of Prof. Dieter Fox at the NVIDIA Robotics Lab. Felix is a recipient of the MIT Presidential Fellowship and the Work of the Future Fellowship in Generative AI at MIT. His research has been recognized with oral and spotlight presentations at CoRL and ICLR, featured on PBS, and is currently exhibited at the MIT Museum.
