Imagine you’re at a new airport, and you need to find your gate. You might use visual cues like signs and landmarks to navigate, but what if you’re in a strange building or country? That’s where humans rely on internal knowledge, like knowing that toilets and showers are usually near bedrooms. But how do robots do it?
Robot Navigation
Researchers have developed various methods for robots to navigate unfamiliar environments. One approach is called CLIP on Wheels (CoW), which uses a robot to explore the closest frontier until the target object is detected using an object detector. Another method is Large Language Model (LLM) based, which processes object detections presented in the form of text to identify the most likely frontier to harbor the target object. However, these methods have limitations.
Visual-Language Fusion Model (VLFM)
Introducing VLFM, a new method that combines visual and language processing to navigate novel environments. Instead of converting visual cues into text before evaluation, VLFM generates semantic value scores directly from RGB observations and text prompts. This approach eliminates the need for remote servers and large amounts of compute, making it more practical for real-world applications.
How Does VLFM Work?
Imagine a robot equipped with a camera and a computer vision model to process visual observations. The robot can also carry a small computer running a language model (such as BERT) to generate text embeddings of object categories. When the robot encounters an unfamiliar environment, it generates a text prompt based on its current visual observation. Then, VLFM compares the text embedding of the target object category with the text embeddings of nearby objects detected by the computer vision model. Finally, the robot navigates directly to the detected target using its navigation system.
Advantages and Limitations
VLFM offers several advantages over traditional methods. Firstly, it eliminates the need for remote servers or large compute resources, making it more practical for real-world applications. Secondly, VLFM can handle unstructured environments with diverse objects and layouts, while still providing accurate navigation. However, VLFM has some limitations. For instance, it may struggle in complex environments with many distractors or when the target object is not easily recognizable from visual cues alone.
Conclusion
In summary, navigating novel environments is a complex task that humans rely on internal knowledge to accomplish. Robots face similar challenges but require more practical and efficient methods. VLFM offers a promising solution by combining visual and language processing to generate semantic value scores directly from RGB observations and text prompts. While it has some limitations, VLFM provides a more practical and robust approach to robot navigation in unfamiliar environments.
