Enhancing Referring Expression Generation with Multimodal Fusion and Visual Guidance

In this paper, researchers aim to enhance the generalization of referring expression comprehension models by boosting the target prompt and introducing multi-modal fusion and visual guidance. The proposed approach leverages a unified context to facilitate target capturing and introduces a new module that fuses visual and linguistic features. Extensive experiments demonstrate the effectiveness of the proposed method, achieving significant improvements in zero-shot performance on various datasets.
To understand how this works, imagine a referring expression as a recipe for cooking a specific dish. The target prompt is like a secret ingredient that helps the model recognize the intended object. By adding this ingredient to the recipe, the model can better grasp (no pun intended) the context and correctly identify the object in the image.
The researchers also introduce multi-modal fusion and visual guidance to further improve performance. Multi-modal fusion combines information from different sources, such as text and images, to create a more robust representation of the object. Visual guidance uses a powerful pre-trained model to leverage spatial relations and visual coherences in the image.
The proposed approach is tested on various datasets, including RefCOCO, RefCOCO+, and ReferItGraspNet-RIS. The results show that boosting the target prompt significantly enhances the model’s ability to generalize to unconstrained textual descriptions. Introducing multi-modal fusion and visual guidance further improves performance, with noticeable improvements over the baseline.
In summary, this paper proposes a novel approach to improve referring expression comprehension by boosting the target prompt and introducing multi-modal fusion and visual guidance. The proposed method leverages a unified context and fuses visual and linguistic features to enhance performance. Experimental results demonstrate the effectiveness of the proposed approach, significantly improving zero-shot performance on various datasets.