Bridging the Modality Gap: Generalized Referring Expression Segmentation

Referring expression segmentation (RES) is a task that involves identifying objects in an image based on natural language instructions. While classic RES algorithms focus on single-target cases, real-world applications often require handling multiple-target and empty-target cases. To address this gap, the Generalized Referring Expression Segmentation (GRES) task was proposed to support more complicated scenarios. GRES tasks require models to handle multi-modal correspondences between images and text prompts, as well as tackle multiple-target and empty-target cases.
The article discusses several recent works on GRES, including the Mimic-it model [24], which uses large language models to reason about segmentation, and the Lisa model [23], which leverages a large transformer to perform reasoning segmentation via language modeling. The authors also highlight the challenges in handling multiple-target and empty-target cases, which are not addressed by classic RES algorithms.
To demystify complex concepts, the article uses analogies to explain how GRES models work. For instance, the authors compare the process of referring expression segmentation to a chef directing a kitchen staff, where the chef provides instructions for different dishes and the staff members coordinate to prepare them. Similarly, in GRES, the model provides instructions for different objects in an image, and the image segmentation is achieved by coordinating multiple tasks.
The article also emphasizes the importance of considering both visual and linguistic features when tackling GRES tasks. The authors note that while visual features can provide important information about object locations, linguistic features can help disambiguate references to multiple objects. They suggest that models should be designed to incorporate both types of features effectively.
In summary, the article provides a comprehensive overview of recent works on GRES and highlights the challenges in handling multiple-target and empty-target cases. The author uses analogies and everyday language to demystify complex concepts, emphasizing the importance of considering both visual and linguistic features when tackling these tasks.