Computer Vision and Geometry Group, ETH Zurich
Talk given by Yann LeCun at ETH Zürich during "Frontiers of Embodied AI".
updated 5 months ago
Authors: Alexey Gavryushin *, Dingxi Zhang *, Zhao Huang *, Alexandros Delitzas, Jiaqi Chen, Ben Ellis, Cedric Zöllner, Manthan Patel, Manuel Kaufmann, Marc Pollefeys, Xi Wang.
Authors: Oussema Dhaouadi, Zuria Bauer, Johannes Meier, Olaf Wysocki, Marc Pollefeys, Daniel Cremers
Authors: Gabriel Tavernini, Elisabetta Fedele, Tiago Novello, Leonidas Guibas, Marc Pollefeys, Francis Engelmann.
Abstract: While recent advances like SuperDec have revitalized superquadrics for compact 3D scene understanding, they remain limited by rigid geometric constraints and lack robustness to partial observations. We present SuperFlex, an enhanced framework expanding the expressive power and applicability of superquadric decompositions. First, we include bending and tapering parameters, enabling high-fidelity representation of complex geometries. Second, a novel loss formulation and optimization scheme refines initial decompositions, significantly improving accuracy. These high-fidelity primitives enable training a robust model for direct prediction from noisy, partial real-world point clouds, a task previously infeasible for methods requiring full point clouds.
Authors: Johannes Meier, Jonathan Michel, Oussema Dhaouadi, Yung-Hsu Yang, Christoph Reich, Zuria Bauer, Stefan Roth, Marc Pollefeys, Jacques Kaiser, Daniel Cremers
We present OpenFrontier, a simple yet effective framework for general language-conditioned robot navigation. It detects and semantically evaluates visual frontiers directly in image space and grounds them into 3D, enabling efficient zero-shot navigation across diverse goals — without dense semantic 3D mapping, policy training, or model fine-tuning.
Paper: arxiv.org/pdf/2603.05377
Project Page: boysun045.github.io/OpenFrontier-Project
Code: github.com/cvg/OpenFrontier
Related work:
FrontierNet: boysun045.github.io/FrontierNet-Project
website: boysun045.github.io/FrontierNet-Project
code: github.com/cvg/FrontierNet
abstract:
Exploration of unknown environments is crucial for autonomous robots; it allows them to actively reason and decide on what new data to acquire for different tasks, such as mapping, object discovery, and environmental assessment. Existing solutions, such as frontier-based exploration approaches, rely heavily on 3D map operations, which are limited by map quality and, more critically, often overlook valuable context from visual cues. This work aims at leveraging 2D visual cues for efficient autonomous exploration, addressing the limitations of extracting goal poses from a 3D map. We propose a visual-only frontier-based exploration system, with FrontierNet as its core component. FrontierNet is a learning-based model that (i) proposes frontiers, and (ii) predicts their information gain, from posed RGB images enhanced by monocular depth priors. Our approach provides an alternative to existing 3D-dependent goal-extraction approaches, achieving a 15% improvement in early-stage exploration efficiency, as validated through extensive simulations and real-world experiments.
website: behretj.github.io/LostAndFound
accepted to the IEEE Robotics and Automation Letters
Recent approaches have successfully focused on the segmentation of static reconstructions, thereby equipping downstream applications with semantic 3D understanding. However, the world in which we live is dynamic, characterized by numerous interactions between the environment and humans or robotic agents. Static semantic maps are unable to capture this information, and the naive solution of rescanning the environment after every change is both costly and ineffective in tracking e.g. objects being stored away in drawers. With Lost & Found we present an approach that addresses this limitation. Based solely on egocentric recordings with corresponding hand position and camera pose estimates, we are able to track the 6DoF poses of the moving object within the detected interaction interval. These changes are applied online to a transformable scene graph that captures object-level relations. Compared to state-of-the-art object pose trackers, our approach is more reliable in handling the challenging egocentric viewpoint and the lack of depth information. It outperforms the second-best approach by 34% and 56% for translational and orientational error, respectively, and produces visibly smoother 6DoF object trajectories. In addition, we illustrate how the acquired interaction information in the dynamic scene graph can be employed in the context of robotic applications that would otherwise be unfeasible: We show how our method allows to command a mobile manipulator through teach & repeat, and how information about prior interaction allows a mobile manipulator to retrieve an object hidden in a drawer.
Leveraging VLM-based affordance prediction to estimate motion primitives for light switch interaction, we achieve up to 84% operation success in real world experiments. We further introduce a specialized dataset containing 715 images as well as a custom detection model for light switch detection. We demonstrate how the framework can facilitate robot learning through physical interaction by having the robot explore the environment and discover previously unknown relationships in a scene graph representation. Lastly, we propose an extension to the framework to accommodate other functional interactions such as swing doors, showcasing its flexibility.
Videos and Code: timengelbracht.github.io/SpotLight
Paper: arxiv.org/abs/2404.12440
Webpage: spot-compose.github.io
In recent years, modern techniques in deep learning and large-scale datasets have led to impressive progress in 3D instance segmentation, grasp pose estimation, and robotics. This allows for accurate detection directly in 3D scenes, object- and environment-aware grasp prediction, as well as robust and repeatable robotic manipulation. This work aims to integrate these recent methods into a comprehensive framework for robotic interaction and manipulation in human-centric environments. Specifically, we leverage 3D reconstructions from a commodity 3D scanner for open-vocabulary instance segmentation, alongside grasp pose estimation, to demonstrate dynamic picking of objects, and opening of drawers. We show the performance and robustness of our model in two sets of real-world experiments including dynamic object retrieval and drawer opening, reporting a 51% and 82% success rate respectively.
website: labelmaker.org
code: github.com/cvg/LabelMaker
Semantic annotations are indispensable to train or evaluate perception models, yet very costly to acquire. This work introduces a fully automated 2D/3D labeling framework that, without any human intervention, can generate labels for RGB-D scans at equal (or better) level of accuracy than comparable manually annotated datasets such as ScanNet. Our approach is based on an ensemble of state-of-the-art segmentation models and 3D lifting through neural rendering. We demonstrate the effectiveness of our LabelMaker pipeline by generating significantly better labels for the ScanNet datasets and automatically labelling the previously unlabeled ARKitScenes dataset.
github: github.com/cvg/hololens_ros
This paper presents a mixed-reality human-robot teaming system. It allows human operators to see in real-time where robots are located, even if they are not in line of sight. The operator can also visualize the map that the robots create of their environment and can easily send robots to new goal positions. The system mainly consists of a mapping and a control module. The mapping module is a real-time multi-agent visual SLAM system that co-localizes all robots and mixed-reality devices to a common reference frame. Visualizations in the mixed-reality device then allow operators to see a virtual life-sized representation of the cumulative 3D map overlaid onto the real environment. As such, the operator can effectively "see through" walls into other rooms. To control robots and send them to new locations, we propose a drag-and-drop interface. An operator can grab any robot hologram in a 3D mini map and drag it to a new desired goal pose. We validate the proposed system through a user study and real-world deployments.
T. Schöps, T. Sattler, M. Pollefeys, "SurfelMeshing: Online Surfel-Based Mesh Reconstruction", ArXiv, 2018 ( arxiv.org/abs/1810.00729 ).
Source code is available at: github.com/puzzlepaint/surfelmeshing
F. Maninchedda, C. Häne, M. Oswald, M. Pollefeys, Face Reconstruction on Mobile Devices Using a Height Map Shape Model and Fast Regularization, Proc Int. Conf. on 3D Vision (3DV), 2016
Lionel Heng, Mathias Bürki, Gim Hee Lee, Paul Furgale, Roland Siegwart, and Marc Pollefeys, Infrastructure-Based Calibration of a Multi-Camera Rig, In Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), 2014
Christian Häne, Lionel Heng, Gim Hee Lee, Alexey Sizov, and Marc Pollefeys, Real-Time Direct Dense Matching on Fisheye Images Using Plane-Sweeping Stereo, In Proceedings of International Conference on 3D Vision (3DV), 2014
Lionel Heng, Bo Li, and Marc Pollefeys, CamOdoCal: Automatic Intrinsic and Extrinsic Calibration of a Rig with Multiple Generic Cameras and Odometry, In Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2013
Gim Hee Lee, Friedrich Fraundorfer, and Marc Pollefeys, Motion Estimation for a Self-Driving Car with a Generalized Camera, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013
C. Häne, T. Sattler, M. Pollefeys, Obstacle Detection for Self-Driving Cars Using Only Monocular Cameras and Wheel Odometry, Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2015
Project webpage: http://cvg.ethz.ch/research/3d-modeling-on-the-go
More information about the research project: http://cvg.ethz.ch/mobile
This video showcases new mobile technology for Interactive 3D Mobile Reconstruction developed at the Computer Vision and Geometry Group from ETH Zürich. It allows to turn a mobile phone into a 3D scanner
Music: Pamgaea by Kevin MacLeod (incompetech.com)


