OneCanvas: 3D Scene Understanding via Panoramic Reprojection @MatthiasNiessner
OneCanvas: 3D Scene Understanding via Panoramic Reprojection  @MatthiasNiessner
Uploaded June 2026 | Updated November 2026, 3 hours ago
Project: baranowskibrt.github.io/onecanvas

Existing approaches to 3D scene understanding in Vision-Language Models (VLMs) either rely on complex, model-specific geometry encoders or large training budgets in pursuit of spatial reasoning. Instead, OneCanvas aggregates patch features from all views onto a single equirectangular panoramic canvas. Each patch is unprojected to a 3D world coordinate using its depth and camera pose, then placed on the canvas at the continuous longitude and latitude of that point as seen from the canvas origin, with no rasterization onto a pixel grid and no aggregation across views.

A 3D position embedding of the patch's metric coordinates is added to its feature, restoring the depth lost when collapsing the world position to an angular canvas coordinate. Patches from all frames thus share one spatial coordinate system with no fusion or major architectural modifications of the backbone. The pretrained VLM consumes this representation as if it were an ordinary image. Because the canvas can be centered on any pose of interest, the same representation directly supports situated reasoning from a specific viewpoint, a common requirement in robotics and embodied AI.

Thanks to this representation, we can also introduce a spatial pretraining curriculum: by procedurally placing patch features of objects, drawn from real images, at chosen 3D world positions on an otherwise empty canvas, we generate on-the-fly supervision spanning a broad range of spatial reasoning tasks, with answer distributions controlled to reduce spatial reasoning shortcuts. OneCanvas achieves state-of-the-art accuracy on SQA3D and VSI-Bench, and generalizes to out-of-distribution data on SPBench, using an order of magnitude less training compute than the strongest competing methods.
OneCanvas: 3D Scene Understanding via Panoramic ReprojectionLinPrim: Linear Primitives for Differentiable Volumetric RenderingTriFlow: Generating Artist-Like 3D Mesh Topology via Nearest-Vertex Vector FieldsTriPlaneNet: An Encoder for EG3D Inversion (WACV24)DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data (CVPR 2020)Avat3r: Large Animatable Gaussian Reconstruction Model for High-fidelity 3D Head AvatarsCVPR 2024 Paper Compilation - TUM Visual Computing Lab & CollaboratorsSynthesia: Native Dubbing DemoScan2CAD: Learning CAD Model Alignment in RGB-D Scans (CVPR 2019 Oral)Introduction to Deep Learning - 1. Introduction (Summer 2020)TUM AI Lecture Series - Computer Vision Startup Trends & Commercializing Research (Evan Nisselson)Deleted video
Matthias Niessner |

OneCanvas: 3D Scene Understanding via Panoramic Reprojection

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER