Distilling On-device Language Models for Robot Planning with Minimal Human Intervention @KumarLabPenn
Distilling On-device Language Models for Robot Planning with Minimal Human Intervention  @KumarLabPenn
Uploaded June 2025 | Updated November 2026, 3 hours ago
aiXiv: arxiv.org/abs/2506.17486
Project page: zacravichandran.github.io/PRISM
Abstract: Large language models (LLMs) provide robots with powerful contextual reasoning abilities and a natural human interface. Yet, current LLM-enabled robots typically depend on cloud-hosted models, limiting their usability in environments with unreliable communication infrastructure, such as outdoor or industrial settings. We present PRISM, a framework for distilling small language model (SLM)-enabled robot planners that run on-device with minimal human supervision. Starting from an existing LLM-enabled planner, PRISM automatically synthesizes diverse tasks and environments, elicits plans from the LLM, and uses this synthetic dataset to distill a compact SLM as a drop-in replacement of the source model. We apply PRISM to three LLM-enabled planners for mapping and exploration, manipulation, and household assistance, and we demonstrate that PRISM
improves the performance of Llama-3.2-3B from 10-20% of GPT-4o’s performance to over 93% — using only synthetic data. We further demonstrate that the distilled planners generalize across heterogeneous robotic platforms (ground and aerial) and diverse environments (indoor and outdoor). We release all software, trained models, and datasets at zacravichandran.github.io/PRISM.
Distilling On-device Language Models for Robot Planning with Minimal Human InterventionSmart Phones Power Flying RobotsAvian Inspired Grasping For Quadrotor Micro Aerial VehiclesPrecision Agriculture at GRASP Lab, Nov 2013 - Dec 2015In-Flight Formation Control for a Team of Fixed-Wing Aerial VehiclesAutomated Creation of Topological Maps in Indoor EnvironmentPenns Falcon robots fast flightTowards Fully Autonomous Visual Inspection of Dark Featureless Dam Penstocks using MAVsCooperative Transportation using Small Quadrotors using Monocular Vision and Inertial SensingMonocular Event-Based Vision for Obstacle Avoidance with a QuadrotorDecentralization of Multiagent Policies by Learning What to CommunicateThe Open Vision Computer : An Integrated Sensing and Compute System for Mobile Robots
Vijay Kumar |

Distilling On-device Language Models for Robot Planning with Minimal Human Intervention

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER