Panoptic Vision-Language Feature Fields @aslteam
Panoptic Vision-Language Feature Fields  @aslteam
Uploaded January 2024 | Updated November 2026, 2 days ago
This is the accompanying video of our IEEE RA-L 2024 paper "Panoptic Vision-Language Feature Fields".

Paper: doi.org/10.1109/LRA.2024.3354624
arXiv: arxiv.org/abs/2309.05448
Project page: ethz-asl.github.io/pvlff
Code: github.com/ethz-asl/pvlff

Abstract:
Recently, methods have been proposed for 3D open-vocabulary semantic segmentation. Such methods are able to segment scenes into arbitrary classes based on text descriptions provided during runtime. In this paper, we propose to the best of our knowledge the first algorithm for open-vocabulary panoptic segmentation in 3D scenes. Our algorithm, Panoptic Vision-Language Feature Fields (PVLFF), learns a semantic feature field of the scene by distilling vision-language features from a pretrained 2D model, and jointly fits an instance feature field through contrastive learning using 2D instance segments on input frames. Despite not being trained on the target classes, our method achieves panoptic segmentation performance similar to the state-of-the-art closed-set 3D systems on the HyperSim, ScanNet and Replica dataset and additionally outperforms current 3D open-vocabulary systems in terms of semantic segmentation. We ablate the components of our method to demonstrate the effectiveness of our model architecture.

Reference:
Haoran Chen, Kenneth Blomqvist, Francesco Milano, and Roland Siegwart, "Panoptic Vision-Language Feature Fields", IEEE Robotics and Automation Letters (RA-L), 2024.

Affiliations:
Autonomous Systems Laboratory, ETH Zurich, Switzerland.
Panoptic Vision-Language Feature FieldsFootball field reconstruction from monocular imagesWatching the Air Rise: Learning-Based Single-Frame Schlieren DetectionAerial Manipulation with an Omnidirectional Flying PlatformLearning to Predict the Wind for Safe Aerial Vehicle PlanningKeep it brief: Scalable creation of compressed localization mapsOn Flying Backwards: Preventing Run-away of Small, Low-speed, Fixed-wing UAVs in Strong WindsVIZARD - Visual Localzation in UP-DriveVisual-inertial teach and repeat powered by Google TangoAttitude- and Cruise Control of a VTOL Tiltwing UAVAdaptive Tank-based Control for Aerial Physical Interaction with Uncertain Dynamic EnvironmentsOnline Synchronization of Building Model for On-Site Mobile Robotic Construction
aslteam |

Panoptic Vision-Language Feature Fields

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER