Fine-tuning Text-to-Image Diffusion Models for Subject-Driven Generation

In this article, the authors aim to improve the quality of text-to-image synthesis using diffusion models. They propose a new approach that leverages a generative model to learn the style of Calvin and Hobbes-style comics, which are often filled with text. The proposed method involves removing the text from the original comics using open-source OCR tools and incorporating the text into the captions using GPT-4 or Flamingo. This allows the diffusion model to learn the style of the comics more accurately.
The authors also suggest including color images in the dataset to improve the model’s understanding of the style. They use a simple coordinate-based cropping technique to extract panels from the black and white images, resulting in approximately 11,033 panels. Each image is accompanied by a meaningful text caption, which is essential for fine-tuning diffusion models.
The proposed method uses an autoencoder model to compress the input image into a smaller 2D latent vector, allowing for efficient denoising and diffusion processes. The decoder model reconstructs the images from the latent vector, ensuring high-quality results. The authors emphasize the importance of incorporating conditioning information, such as class labels or semantic maps, to handle flexible conditioning for image generation.
In summary, this article presents a novel approach to improving text-to-image synthesis using diffusion models. By leveraging GPT-4 and Flamingo to incorporate text into the captions and including color images in the dataset, the proposed method demonstrates improved performance in capturing the style of Calvin and Hobbes-style comics.