Continuous 3D Perception Model with Persistent State

We introduce CUT3R, a unified framework for continuous 3D perception using a recurrent, stateful transformer. Given a stream of RGB images—whether video or unordered photo collections—our model incrementally updates a persistent scene state and generates metric-scale pointmaps and camera poses online. This enables dense, online 3D reconstruction without assuming static scenes or known camera parameters.

Our model leverages learned priors to handle sparse inputs, degenerate motion, and dynamic content, while supporting inference of unobserved scene regions via virtual camera queries. Unlike tabula rasa methods like SfM/SLAM or NeRF, CUT3R builds a coherent 3D world model over time. The framework is simple, general, and effective for monocular depth, pose estimation, and reconstruction tasks across diverse scenes.

Figure 1. Continuous 3D Perception. Given a stream of RGB images as input, our approach enables dense 3D reconstruction in an online, continuous manner, estimating both camera parameters and dense 3D geometry with each incoming frame. Our framework supports various 3D tasks, processes inputs from video sequences and sparse photo collections, and can handle both static and dynamic scenes.

Traditional 3D reconstruction methods like SfM and SLAM are tabula rasa, requiring known camera parameters and failing under sparse or degenerate inputs. Learning-based approaches such as NeRF and 3D Gaussian Splatting generate geometry but need predefined poses. DUSt3R predicts metric pointmaps for image pairs but requires offline global alignment and cannot update incrementally.

Spann3R supports variable-length input with a spatial memory but only acts as a cache, not enabling inference of unseen regions. MonST3R extends DUSt3R to dynamic scenes via finetuning but still relies on post-processing alignment. Our method is the first to combine online, continuous updates, metric reconstruction, and generalizable inference of unobserved structures—all in a single feedforward, recurrent architecture.

CUT3R is trained on 32 diverse datasets covering synthetic and real-world scenes, including static and dynamic environments. These include ARKitScenes, ScanNet++, CO3Dv2, TartanAir, Waymo, MegaDepth, and MapFree. The dataset suite spans indoor, outdoor, object-centric, and large-scale 3D scenes. Each dataset provides varying annotations: metric depth, camera poses, or both.

We extended some datasets (e.g., MapFree, DL3DV) with MVS-derived depth maps. Our training leverages partial annotations—e.g., camera-only or single-view data—through curriculum learning, enhancing robustness across domains. This breadth ensures the model generalizes to real-world variations.

CUT3R uses a ViT-Large encoder to token-encode input images, which interact with a persistent state via two interconnected transformers—state-update and state-readout. The image tokens update the state while simultaneously retrieving prior context. From the enriched features, we predict two metric pointmaps (in self and world frames) and camera pose using DPT and MLP heads, respectively.

A pose token captures ego-motion. For unseen view inference, a raymap query (origin + direction per pixel) is encoded and read from the state without updating it. Output includes pointmaps and color predictions, enabling metric-scale reconstruction and hallucination of unobserved regions.

Figure 3. Method Overview . Our method performs online dense 3D reconstruction from a stream of images (video frames or a photo collection) by using a persistent state. Each input image is encoded into visual tokens via a shared-weight ViT encoder. These tokens interact with state tokens, where state update integrates the current image into the state, and state readout retrieves the past context stored in the state for predictions. Both processes occur simultaneously through two interconnected ViT decoders. Outputs include pointmaps in world and camera frames (only world pointmaps are shown) and the camera-to-world transformation. On the right, we demonstrate our method's ability to predict unseen views: given a query camera (as a raymap), it reads information from the state to predict its corresponding pointmap, even for unobserved regions. For these readouts, we do not update the state. The hallucinated pointmap is highlighted with a blue border.

We evaluate on monocular depth (KITTI, NYU-v2), video depth (Sintel, Bonn, KITTI), camera pose (Sintel, TUM-dynamics, ScanNet), and 3D reconstruction (7-Scenes, NRGBD). CUT3R achieves state-of-the-art or competitive results, outperforming DUSt3R, MASt3R, MonST3R, and Spann3R.

On video depth, it improves over DUSt3R by 8.6% δ<1.25 on KITTI and 25× faster than optimization-based methods. On pose estimation, it ranks best among online methods, especially on dynamic scenes. 3D reconstruction achieves 0.113 Acc on 7-Scenes, comparable to offline methods while being fully online and 25× faster.

Table 2. Video Depth Evaluation . We report scale-invariant depth and metric depth accuracy on Sintel, Bonn, and KITTI datasets. Methods requiring global alignment are marked 'GA', while 'Optim.' and 'Onl.' indicate optimization-based and online methods, respectively. We also report the FPS on KITTI dataset using 512 × 144 image resolution for all methods on an A100 GPU, except Spann3R which only supports 224 × 224 inputs. We present a subset of baselines here; please refer to the supplementary material for full comparisons.

Table 3. Evaluation on Camera Pose Estimation on Sintel [12], TUM-dynamic [89], and ScanNet [19] datasets. Our method achieves the best overall performance among all online methods.

Table 4. 3D reconstruction comparison on 7-Scenes [83] and NRGBD [4] datasets. While operating online, our method achieves competitive performance, on par with and even surpassing offline methods that employ global alignment.

CUT3R powers real-time 3D-aware systems: robotics navigation, AR/VR scene tracking, and autonomous vehicles. It enables dense reconstruction from casual video clips (e.g., user-recorded videos). The ability to query unobserved regions allows simulating missing views in autonomous driving scenarios or reconstructing partially occluded cities.

In AR, our model can hallucinate unseen building facades or furniture from a single photo. For indoor scene editing, it supports dynamic reconstruction during navigation. The system works on low-overlap collections—ideal for internet-scale data curation or reconstructing damaged heritage sites from scattered photos.

Figure 5. State Update Analysis. Compared to online, revisiting incorporates global context which improves overall reconstruction results, especially in the highlighted regions.

Figure 6. Inferring New Structure via. State Readout . From top to bottom: the input image; the ground truth (GT) image, used to query the state via its camera parameters (note: GT image is not given to the model); the depth map from the predicted pointmap; the pointmap prediction of the input image alone; and the pointmap combined with the predicted pointmap in a shared coordinate frame.