Per-scene optimization methods such as LangSplat, OpenGaussian, Dr. Splat and InstanceGaussian cannot be generalized to novel scenes without additional optimization.
EmbodiedSplat
EmbodiedSplat
CVPR 2026
National University of Singapore
TLDR
Build and understand at once! By taking over 300 streaming images, EmbodiedSplat reconstructs whole-scene open-vocabulary 3DGS in an online manner at up to 5–6 FPS per-frame processing time. The reconstructed scene supports diverse perception tasks: open-vocabulary 3D semantic segmentation, 2D-rendered semantic segmentation, and novel-view color synthesis with depth rendering.
Motivation
3D perception module for embodied agents should fulfill the 5 conditions.
However, existing semantic 3DGS methods are not applicable to embodied scenarios because:
Per-scene optimization methods such as LangSplat, OpenGaussian, Dr. Splat and InstanceGaussian cannot be generalized to novel scenes without additional optimization.
Feed-forward methods such as LSM and SIU3R propose language-embedded feed-forward semantic 3DGS. However, they are restricted to offline setting and only address only two or a few input views.
Auto-encoder / decoder from LangSplat or PQ-Index from Dr. Splat both require an additional pretraining stage and suffer from severe information loss due to dimensionality reduction. Per-scene optimized codebook from OpenGaussian or InstanceGaussian can't be adopted to generalizable setting.
Contributions
Building on top of pretrained feed-forward 3DGS model (FreeSplat++), we propose the 3DGS-based perception model designed for embodied scenarios where the model supports online semantic 3DGS reconstruction in near real-time.
One of the key innovations of EmbodiedSplat lies in its memory-efficient representation of per-Gaussian CLIP features. We propose novel Online Sparse Coefficient Field with CLIP global codebook which offers several advantages over existing CLIP compressors:
CLIP features derived from 2D images exhibit strong open-vocabulary capability while lacking the 3D geometry prior. Therefore, we complement 3D CLIP features which are produced by aggregating the feature point cloud of 3DGS through 3D U-Net and temporal-aware memory adapter. Both using 2D and 3D CLIP features enables mutual compensation where 2D features offer strong semantic capability while 3D features complement 3D geometry prior.
Overall, our contributions are three-fold:
Novel framework enabling online, whole-scene reconstruction for language-embedded 3DGS with up to 5–6 FPS inference speed — the first feed-forward approach at this scale.
Sparse Coefficient Field with a CLIP Global Codebook stores per-Gaussian language embeddings compactly while preserving the full semantic generalizability of CLIP.
Combination of 2D CLIP Features with rich semantic capabilities and 3D CLIP Features with geometric prior, enabling mutual compensation between semantics and 3D geometry.
Comparison with Existing Methods
Only EmbodiedSplat satisfies all five criteria required for embodied 3D perception.
| Method | Online | Near Real-time | Feed-Forward | Whole-scene Recon | Open-vocabulary |
|---|---|---|---|---|---|
| LangSplat-v1, v2 | ✗ | ✗ | ✗ | ✓ | ✓ |
| OpenGaussian | ✗ | ✗ | ✗ | ✓ | ✓ |
| InstanceGaussian | ✗ | ✗ | ✗ | ✓ | ✓ |
| Dr. Splat | ✗ | ✗ | ✗ | ✓ | ✓ |
| LSM | ✗ | ✓ | ✓ | ✗ | ✓ |
| SIU3R | ✗ | ✓ | ✓ | ✗ | ✓ |
| Online-LangSplat | ✓ | 🤔 | ✗ | ✓ | ✓ |
| EA3D | ✓ | 🤔 | ✗ | ✓ | ✓ |
| EmbodiedSplat | ✓ | ✓ | ✓ | ✓ | ✓ |
Video Demo
Enjoy the visualization of our online 3DGS reconstruction with language embeddings.
Interactive Demo
Loading may take a moment — the 3DGS file is large.
Method
We endow feed-forward 3DGS with semantic understanding by binding two types of CLIP features: ① 2D Semantic Features — attached to each Gaussian via the Sparse Coefficient Field with CLIP Global Codebook, reducing memory while preserving CLIP generalizability; and ② 3D Geometric-aware Features — produced by aggregating the feature point cloud of 3DGS through a 3D U-Net and temporal-aware memory adapter. Their mutual compensation yields superior understanding over existing baselines.
Citation
@inproceedings{lee2026embodiedsplat,
title={EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding},
author={Lee, Seungjun and Wang, Zihan and Wang, Yunsong and Lee, Gim Hee},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={23774--23784},
year={2026}
}