Deep Learning for Image Captioning: A Survey of Attacks and Defenses

Image captioning is a task that involves generating natural language descriptions for images. In this article, the authors explore how image captioning can be approached as a ranking task, where the goal is to rank images based on their relevance to a given description. This approach allows researchers to use data and models designed for image ranking tasks to improve image captioning performance.
The authors discuss several key concepts in image captioning, including features, models, and evaluation metrics. They explain that features are the characteristics of an image that can be used to describe it, while models are the algorithms that generate captions based on these features. Evaluation metrics, such as BLEU score, measure how well a model performs in generating accurate and relevant captions.
The authors highlight some challenges associated with image captioning, including the need for large amounts of training data to achieve good performance and the risk of backdoors being introduced into the models through unknown-source third-party data. A backdoor is a vulnerability in a model that can be exploited by an attacker to manipulate its output, such as generating captions that are not accurate or relevant for a given image.
To overcome these challenges, the authors propose framing image description as a ranking task, which allows researchers to use data and models designed for image ranking tasks to improve image captioning performance. They also discuss several techniques for improving image captioning performance, including using pre-trained language models and incorporating additional information about the image, such as its location or context.
Overall, this article provides a comprehensive overview of the challenges and opportunities in image captioning, and demonstrates how framing the task as a ranking task can help researchers to overcome these challenges and improve the performance of image captioning models.