3D Foundation Models

Introduction

Foundation models have made remarkable progress in both natural language processing (NLP) and computer vision, transforming these domains with their exceptional accuracy and generalization capabilities. In recent years, impressive improvements in 2D vision tasks have been achieved through the introduction of these models, setting new standards for performance and adaptability.

The secret behind the success of foundation models lies in the scaling law: by significantly increasing model parameters and training data, these models exhibit emerging properties that were previously unattainable.

This revolutionary approach, which has already reshaped NLP and 2D computer vision, is now being applied to create foundation models for 3D reconstruction, unlocking unprecedented potential for applications in AR/VR, Robotics and GenAI.

The rise of 3D foundation models marks a transformative moment for computer vision, enabling machines to perceive depth, spatial relationships, and geometric structures. This advancement is critical for addressing real-world challenges in dynamic and complex 3D environments, paving the way for intelligent systems to thrive in domains where spatial understanding is essential.

What Are Foundation Models?

Foundation models are large-scale, pre-trained neural networks designed to generalize across a wide range of tasks. Initially popularized in natural language processing (NLP), these models have since expanded into computer vision, where they extract universal representations from vast datasets, enabling fine-tuning for specific tasks with minimal data.

However, the development of foundation models in NLP and vision follows distinct trajectories due to fundamental differences in data structure, learning paradigms, and computational demands.

Key Differences Between NLP and Vision Foundation Models

Data Modality and Structure

NLP: Text data is sequential and discrete, making it ideal for self-supervised learning tasks like next-word prediction. The structured nature of language allows models to capture semantic relationships and contextual dependencies effectively.
Vision: Image data is continuous and spatial, requiring models to interpret pixel-level information and hierarchical features. Unlike NLP, self-supervised learning in vision is more complex, often relying on pretext tasks like masked image modeling or embedding contrastive learning.

Learning Paradigm

NLP: Foundation models in NLP are primarily driven by self-supervised learning (e.g., next-word prediction), allowing them to learn meaningful representations from vast amounts of unlabeled text without human annotation.
Vision: While self-supervised learning is advancing, most vision foundation models still depend on supervised or semi-supervised learning due to the complexity of visual data and the challenges of designing effective pretext tasks.

Scalability and Computational Complexity

NLP: Text data is lightweight, enabling large-scale training on massive corpora like Wikipedia or Common Crawl, allowing models to generalize effectively.
Vision: Image and 3D data are computationally intensive, requiring significantly more storage and processing power. Training vision foundation models, especially in 3D, involves handling large-scale, multi-modal datasets, which increases computational demands.

As foundation models continue to advance in NLP and 2D vision, their expansion into 3D AI presents new challenges and opportunities. Developing models that effectively capture depth, geometry, and spatial relationships requires innovative approaches to efficiently process large-scale 3D data while leveraging the strengths of self-supervised learning. Despite these challenges, recent breakthroughs have demonstrated remarkable progress, pushing the boundaries of what’s possible in 3D AI. In the next section, we explore some of the latest advancements that are shaping the future of 3D foundation models.

Recent Advances in 3D Foundation Models

Recent breakthroughs in 3D foundation models are redefining how machines perceive and reconstruct the 3D world. In this section, we explore key advancements that push the boundaries of geometric learning and spatial understanding.

DUSt3R: Geometric 3D Vision Made Easy

DUSt3R introduces a novel method for 3D reconstruction that eliminates the need for explicit camera calibration. Instead, it learns to predict pointmaps—dense 2D fields of 3D points—directly from image pairs. The key innovations include:

Pointmap Representation – Instead of estimating depth, DUSt3R directly predicts a 3D position for each pixel, enabling implicit 3D reconstruction without camera parameters.
Dual-View Processing – The model takes two RGB images as input and estimates pointmaps in a shared coordinate frame, allowing implicit triangulation for improved 3D consistency.
Transformer-Based Network – A Siamese Vision Transformer (ViT) encoder extracts image features, while cross-attention mechanisms in the decoder facilitate geometric reasoning.

End-to-End Learning – The model is trained to infer scene geometry directly from image pairs, bypassing traditional feature matching and explicit depth estimation, leading to a more robust and flexible reconstruction pipeline.

The DUSt3R architecture shows how two views of a scene (I1 , I2) are first encoded in a Siamese manner with a shared ViT encoder. The resulting token representations F1 and F2 are then passed to two transformer decoders that constantly exchange information via cross-attention. Finally, two regression heads output the two corresponding pointmaps and associated confidence maps. Importantly, the two pointmaps are expressed in the same coordinate frame of the first image I1. The network is trained using a simple regression loss.

The model was trained in a fully supervised manner, leveraging large-scale ground-truth 3D maps and multi-view correspondence data. It was trained on a diverse mixture of eight datasets, spanning indoor, outdoor, synthetic, and real-world scenes. The dataset was curated to 8.5M image pairs, ensuring strong generalization across 3D environments.

DUSt3R streamlines 3D vision by unifying multiple tasks within a single pipeline. Instead of relying on separate models for camera calibration, pose estimation, depth estimation, and 3D reconstruction.

DepthAnything: Advancing Monocular Depth Estimation

Monocular depth estimation, the task of predicting scene depth from a single image, has seen remarkable improvements through foundation models. Depth Anything follows a data-centric approach to achieve high generalization across diverse scenarios.

DepthAnything: Unleashing the Power of Large-Scale Unlabeled Data

DepthAnything V1 pioneered large-scale semi-supervised learning for monocular depth estimation by leveraging pseudo-labeling techniques. Given the scarcity of high-quality ground-truth depth data, the model adopted key steps of a pseudo-labeling approach:

Teacher Model Pretraining: The model (teacher) was first trained on 1.5M labeled images collected from six diverse datasets.
Pseudo-Label Generation: The trained teacher model was then used to generate pseudo-depth labels for an additional 62M unlabeled images, vastly expanding the dataset.
Student Model Training: A final student model was trained using a combination of ground-truth labeled and pseudo-labeled data, enabling robust generalization.

This strategy allowed DepthAnything to achieve impressive generalization, demonstrating strong zero-shot capabilities across diverse scenes. Furthermore, fine-tuning the model with metric depth supervision led to state-of-the-art performance, highlighting its learned robust depth prior.

DepthAnything V2: The Power of Synthetic Data with Pseudo labeling

DepthAnything V2 builds upon its predecessor by refining the training strategy and scaling up model capacity. Unlike V1, which used real-image supervision for pseudo-labeling, V2 makes three key advancements:

Synthetic-Only Training for the Teacher Model – V2 removes real ground-truth depth supervision and instead pretrains the teacher model entirely on synthetic images(595K images), enabling it to capture fine-grained depth details without the inconsistencies of real-world annotations.
Scaling Up the Teacher Model – A larger and more capable teacher is used, improving depth quality and robustness.
Large-Scale Pseudo-Labeling for the Student Model – Training solely on synthetic images introduces a domain gap when transferring to real-world scenes. To bridge this gap, the teacher model generates pseudo-depth labels for 62M+ real unlabeled images, and the student model is trained exclusively on this pseudo-labeled real dataset. This approach improves scene diversity and enhances generalization to real-world depth estimation.

This new approach removes dependency on real-world annotations, enhances generalization across diverse scenes, and produces finer and more robust depth predictions.

DepthAnything V2 represents a major step toward a foundational monocular depth model.

What’s Next?

The evolution of 3D foundation models points toward an AI-driven future, but key challenges remain:

Dependence on Fully Supervised Learning
- Current models rely on either real labels (which are expensive and often inaccurate) or synthetic data (which introduces a domain gap).
- The self-supervised learning revolution remains an open challenge for 3D perception.
Bridging the Synthetic-to-Real Domain Gap
- While synthetic data provides high-quality labels, models must generalize to real-world environments without domain-specific biases.
Achieving Camera Model and Lens Distortion Invariance
- To be truly universal, 3D models must handle different camera types, focal lengths, and distortions without requiring dataset-specific adaptations.

As research continues to refine these models, the next frontier is bringing them from research labs to real-world applications—ultimately making 3D understanding more accessible, scalable, and adaptable across industries.