A Review of Open-Vocabulary Semantic Mapping and Navigation with Foundation Models for Mobile Robots

Engineering notes

Engineering notes will be added by the aipentium editorial team.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为大语言模型、生成式AI、ChatGPT相关技术、计算机视觉、深度学习等高价值论文补充中文说明。

Original abstract

Autonomous mobile robots that coexist with humans must construct not only geometric maps but also semantic maps that can be accessed through natural language. Conventional semantic mapping has mainly focused on assigning labels from predefined closed vocabularies to metric maps, limiting its ability to handle novel objects, open-ended linguistic expressions, and flexible human-robot interaction. Recent advances in large-scale foundation models, particularly LLMs and VLMs, have accelerated research on open-vocabulary semantic mapping. In parallel, generative 3D representations such as neural radiance fields and 3D Gaussian splatting have enabled dense, continuous spatial representations associated with language-derived features. Together, these developments allow robots to acquire spatial semantic representations that connect perception, language, and action. This paper reviews this rapidly evolving field through a four-part taxonomy: (i) fusion of semantic features into 3D metric maps; (ii) object-centric open-vocabulary representations; (iii) hierarchical scene graph representations; and (iv) continuous generative 3D language fields. We also revisit the history of open-vocabulary semantic mapping and provide an overview of foundation model-based navigation using language-accessible maps, ranging from object-goal navigation to LLM-based hierarchical task planning. Finally, we introduce evaluation datasets, simulators, robot platforms, and evaluation metrics, and summarize seven open challenges.

Links and sources

Comments