MatchLab/UniScene3D

RGB-Pointmap Pretraining for Unified 3D Scene Understanding

UniScene3D is a transformer-based encoder that learns unified scene representations from multi-view colored pointmaps, jointly modeling image appearance and geometry. It extends pretrained CLIP models to learn representations that effectively combine complementary information from images and pointmaps, generalizing across diverse 3D scene understanding tasks.

• Project Page: https://yebulabula.github.io/UniScene3D/
• GitHub Repository: https://github.com/Yebulabula/UniScene3D
• Paper: https://huggingface.co/papers/2604.02546

Key Features

• Unified Representation: Jointly encodes geometry and appearance from multi-view colored pointmaps within a single ViT encoder.
• Novel Training Objectives: Introduces cross-view geometric alignment and grounded view alignment to enforce geometric and semantic consistency.
• Versatile Performance: Demonstrates state-of-the-art performance in zero-shot, few-shot, and task-specific fine-tuning settings for tasks like viewpoint grounding, scene retrieval, and 3D VQA.

Citation

If you find this work useful, please cite:

@article{mao2026rgb,
  title={RGB-Pointmap Pretraining for Unified 3D Scene Understanding},
  author={Mao, Ye and Luo, Weixun and Huang, Ranran and Jing, Junpeng and Mikolajczyk, Krystian},
  journal={The 19th European Conference on Computer Vision},
  year={2026}
}