Uploaded January 2024 | Updated November 2026, 2 days ago
This is the accompanying video of our IEEE RA-L 2024 paper "Panoptic Vision-Language Feature Fields".
Paper: doi.org/10.1109/LRA.2024.3354624
arXiv: arxiv.org/abs/2309.05448
Project page: ethz-asl.github.io/pvlff
Code: github.com/ethz-asl/pvlff
Abstract:
Recently, methods have been proposed for 3D open-vocabulary semantic segmentation. Such methods are able to segment scenes into arbitrary classes based on text descriptions provided during runtime. In this paper, we propose to the best of our knowledge the first algorithm for open-vocabulary panoptic segmentation in 3D scenes. Our algorithm, Panoptic Vision-Language Feature Fields (PVLFF), learns a semantic feature field of the scene by distilling vision-language features from a pretrained 2D model, and jointly fits an instance feature field through contrastive learning using 2D instance segments on input frames. Despite not being trained on the target classes, our method achieves panoptic segmentation performance similar to the state-of-the-art closed-set 3D systems on the HyperSim, ScanNet and Replica dataset and additionally outperforms current 3D open-vocabulary systems in terms of semantic segmentation. We ablate the components of our method to demonstrate the effectiveness of our model architecture.
Reference:
Haoran Chen, Kenneth Blomqvist, Francesco Milano, and Roland Siegwart, "Panoptic Vision-Language Feature Fields", IEEE Robotics and Automation Letters (RA-L), 2024.
Affiliations:
Autonomous Systems Laboratory, ETH Zurich, Switzerland.
This is the accompanying video of our IEEE RA-L 2024 paper "Panoptic Vision-Language Feature Fields".
Paper: doi.org/10.1109/LRA.2024.3354624
arXiv: arxiv.org/abs/2309.05448
Project page: ethz-asl.github.io/pvlff
Code: github.com/ethz-asl/pvlff
Abstract:
Recently, methods have been proposed for 3D open-vocabulary semantic segmentation. Such methods are able to segment scenes into arbitrary classes based on text descriptions provided during runtime. In this paper, we propose to the best of our knowledge the first algorithm for open-vocabulary panoptic segmentation in 3D scenes. Our algorithm, Panoptic Vision-Language Feature Fields (PVLFF), learns a semantic feature field of the scene by distilling vision-language features from a pretrained 2D model, and jointly fits an instance feature field through contrastive learning using 2D instance segments on input frames. Despite not being trained on the target classes, our method achieves panoptic segmentation performance similar to the state-of-the-art closed-set 3D systems on the HyperSim, ScanNet and Replica dataset and additionally outperforms current 3D open-vocabulary systems in terms of semantic segmentation. We ablate the components of our method to demonstrate the effectiveness of our model architecture.
Reference:
Haoran Chen, Kenneth Blomqvist, Francesco Milano, and Roland Siegwart, "Panoptic Vision-Language Feature Fields", IEEE Robotics and Automation Letters (RA-L), 2024.
Affiliations:
Autonomous Systems Laboratory, ETH Zurich, Switzerland.


![Aerial Manipulation with an Omnidirectional Flying Platform
This video showcases our current research to address the challenges of aerial manipulation with omnidirectional flying robots at ETH Zurichs Autonomous systems lab. This topic connects several different topics of active research at our institute, including design and control of omnidirectional aerial manipulators, planning frameworks, and grasp detection.
For further details on the content of this video, links to the related research papers are listed below.
Paper References
[1] Allenspach, M., Bodie, K., Brunner, M., Rinsoz, L., Taylor, Z., Kamel, M., Siegwart, R. and Nieto, J., 2020. Design and optimal control of a tiltrotor micro-aerial vehicle for efficient omnidirectional flight., The International Journal of Robotics Research, 39(10-11), pp.1305-1325.
link: https://arxiv.org/pdf/2003.09512.pdf
[2] Zhang, W., Brunner, M., Ott, L., Kamel, M., Siegwart, R. and Nieto, J., 2020. Learning dynamics for improving control of overactuated flying systems., IEEE Robotics and Automation Letters, 5(4), pp.5283-5290.
link: https://arxiv.org/pdf/2006.13153.pdf
[3] Zhang, W., Ott, L., Tognon, M., Siegwart, R. and Nieto, J., 2021. Active Model Learning using Informative Trajectories for Improved Closed-Loop Control on Real Robots. arXiv preprint arXiv:2101.08100.
link: https://arxiv.org/pdf/2101.08100.pdf
[4] Bodie, K., Brunner, M., Pantic, M., Walser, S., Pfändler, P., Angst, U., Siegwart, R. and Nieto, J., 2019. An omnidirectional aerial manipulation platform for contact-based inspection., Robotics: Science and System XV, 15.
link: https://arxiv.org/pdf/1905.03502.pdf
[5] Bodie, K., Tognon, M. and Siegwart, R. Dynamic End Effector Tracking with an Omnidirectional Parallel Aerial Manipulator. (under review, link to follow)
[6] Breyer, M., Chung, J.J., Ott, L., Siegwart, R. and Nieto, J., 2021. Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter., arXiv preprint arXiv:2101.01132.
link: https://arxiv.org/pdf/2101.01132.pdf
[7] Pantic, M., Ott, L., Cadena, C., Siegwart, R. and Nieto, J., 2021. Mesh Manifold based Riemannian Motion Planning for Omnidirectional Micro Aerial Vehicles., IEEE Robotics and Automation Letters, 6(3), pp.4790-4797.
link: https://arxiv.org/pdf/2102.10313.pdf
[8] Allenspach, M., Lawrance, N. , Tognon, M. and Siegwart, R. 6 DOF Teleoperation of an Omnidirectional Aerial Vehicle. (under review, link to follow) Aerial Manipulation with an Omnidirectional Flying Platform](https://i.ytimg.com/vi/njlyQnRwNLs/mqdefault.jpg)




![Visual-inertial teach and repeat powered by Google Tango
Visual-Inertial Teach and Repeat for Aerial Inspection
M Fehr, T Schneider, M Dymczyk, J Sturm, R Siegwart
Arxiv: https://arxiv.org/abs/1803.09650
Submitted to:
Workshop on Aerial Robotic Inspection and Maintenance
International Conference on Robotics and Automation 2018
Related Publications:
Real-time visual-inertial mapping, re-localization and planning onboard MAVs in unknown environments
M Burri, H Oleynikova, MW Achtelik, R Siegwart
A robust and modular multi-sensor fusion approach applied to mav navigation
S Lynen, MW Achtelik, S Weiss, M Chli, R Siegwart
Linear vs Nonlinear MPC for Trajectory Tracking Applied to Rotary Wing Micro Aerial Vehicles
M Kamel, M Burri, R Siegwart
Related Videos:
https://www.youtube.com/watch?v=tuUMwcTJx8s
https://www.youtube.com/watch?v=Ot76i9p2ZZo
https://www.youtube.com/watch?v=lrGSwAPzMOQ
Contact:
marius.fehr[at]mavt.ethz.ch
thomas.schneider[at]mavt.ethz.ch Visual-inertial teach and repeat powered by Google Tango](https://i.ytimg.com/vi/pDlQXsOrgI4/mqdefault.jpg)


