WorldAgents: Can Foundation Image Models be Agents for 3D World Models? @MatthiasNiessner
WorldAgents: Can Foundation Image Models be Agents for 3D World Models?  @MatthiasNiessner
Uploaded March 2026 | Updated November 2026, 12 hours ago
ziyaerkoc.com/worldagents

Given the remarkable ability of 2D foundation image models to generate high-fidelity outputs, we investigate a fundamental question: do 2D foundation image models inherently possess 3D world model capabilities? To answer this, we systematically evaluate multiple state-of-the-art image generation models and Vision-Language Models (VLMs) on the task of 3D world synthesis.

To harness and benchmark their potential implicit 3D capability, we propose an agentic framing to facilitate 3D world generation. Our approach employs a multi-agent architecture: a VLM-based director that formulates prompts to guide image synthesis, a generator that synthesizes new image views, and a VLM-backed two-step verifier that evaluates and selectively curates generated frames from both 2D image and 3DGS reconstruction space.

Crucially, we demonstrate that our agentic approach provides coherent and robust 3D reconstruction, producing output scenes that can be explored by rendering novel views. Through extensive experiments across various foundation models, we demonstrate that 2D models do indeed encapsulate a grasp of 3D worlds. By exploiting this understanding, our method successfully synthesizes expansive, realistic, and 3D-consistent worlds.
WorldAgents: Can Foundation Image Models be Agents for 3D World Models?WorldExplorer: Towards Generating Fully Navigable 3D Scenes (SIGGRAPH Asia25)OneCanvas: 3D Scene Understanding via Panoramic ReprojectionLinPrim: Linear Primitives for Differentiable Volumetric RenderingTriFlow: Generating Artist-Like 3D Mesh Topology via Nearest-Vertex Vector FieldsTriPlaneNet: An Encoder for EG3D Inversion (WACV24)DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data (CVPR 2020)Avat3r: Large Animatable Gaussian Reconstruction Model for High-fidelity 3D Head AvatarsCVPR 2024 Paper Compilation - TUM Visual Computing Lab & CollaboratorsSynthesia: Native Dubbing DemoScan2CAD: Learning CAD Model Alignment in RGB-D Scans (CVPR 2019 Oral)Introduction to Deep Learning - 1. Introduction (Summer 2020)
Matthias Niessner |

WorldAgents: Can Foundation Image Models be Agents for 3D World Models?

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER