Metric-semantic map representation for natural-language-based navigation

Abstract

The widespread of Large Language Models has sparked a rapid development of general embodied Artificial Intelligence (AI). Many methods to integrate LLMs with robot agent for downstream tasks like manipulation or navigation have been proposed. LLMs have demonstrated exceptionally strong capabilities in understanding, planning, and reasoning to work as the brain of a robot agent. However, grounding the model and deploying it in real world remain challeng- ing. Motivated by the need for more adaptable and intelligent robotic navigation systems, we present a novel and efficient system designed to complete naviga- tion tasks under natural language instructions. We introduce a GPU-accelerating volumetric semantic map for scene representation and environment perception. We use prompts that combine world perception and robot capabilities, enabling LLMs to generate executable plans directly as programs, and build a close- loop robot control framework that enhances task planning and error manage- ment through an integrated feedback structure at both high and low levels. Our framework demonstrates state of the art success rates in zero-shot goal naviga- tion tasks in real-world experiments.