Unsupervised Audio-Caption Alignment via Correspondence Learning

In this article, the authors propose a novel approach to weakly supervised text-to-audio grounding, which learns correspondences between individual sound events and textual phrases. The proposed method, referred to as ReferItGame, leverages contrastive learning and pooling strategies to train a weakly supervised model that can generate high-quality audio embeddings for given captions.
The authors begin by discussing the challenges of weakly supervised text-to-audio grounding, where only a limited number of manual annotations are available for training. They then introduce ReferItGame, which combines a contrastive learning objective with pooling strategies to learn a robust representation of audio and text.
The key idea behind ReferItGame is to train the model to predict whether two pieces of input, either an audio clip or a caption, belong to the same object or not. This is achieved through a self-supervised learning approach, where the model is trained on a large dataset of unlabelled audio clips and their corresponding captions. By maximizing the similarity between the model’s prediction and the true label, the model learns to generate high-quality audio embeddings that correspond to the textual phrases.
The authors evaluate the proposed method on several benchmark datasets and show that ReferItGame outperforms existing weakly supervised methods in terms of both audio quality and correlation with manual annotations. They also demonstrate the versatility of their approach by applying it to various tasks, including speech recognition and music classification.
In conclusion, the article presents a novel approach to weakly supervised text-to-audio grounding that leverages contrastive learning and pooling strategies to train a robust representation of audio and text. The proposed method, ReferItGame, shows promising results in improving the quality of generated audio embeddings and has the potential to enable more efficient and effective speech recognition and music analysis tasks in the future.