Synthesizing High-Quality 4D Scenes from Text with a Hybrid Score Distillation Sampling Procedure

In this article, we present a novel approach to generating high-quality 4D scenes from text prompts using a hybrid score distillation sampling procedure. Our method alleviates a three-way tradeoff between the quality of appearance, 3D structure, and motion in text-to-4D generation, which is a common challenge in this field. We demonstrate the effectiveness of our approach through user studies and compare it to existing methods.
To understand how our method works, let’s first consider the challenges of text-to-4D generation. Traditional methods use score distillation sampling (SDS), which involves rendering an image of a 3D scene, adding noise, and then using a pre-trained diffusion model to denoise the image. However, this process can lead to a tradeoff between the quality of appearance, 3D structure, and motion in the generated scene.
To overcome this challenge, we propose a hybrid approach that combines the strengths of different generative models. We use a combination of text-to-image (T2I), 3D-aware T2I, and text-to-video (T2V) models to generate 4D scenes from text prompts. By leveraging the unique capabilities of each model, we can create high-quality 4D scenes that are not only visually realistic but also capture the intended motion and structure.
Our approach involves three stages of optimization: pre-training a T2I model on a large dataset, fine-tuning a 3D-aware T2I model on a smaller dataset, and using a T2V model to generate the final 4D scene. Each stage requires approximately 20-80 GB of VRAM and around 2-19 hours of compute time, depending on the optimization stage.
We evaluate our method through user studies and compare it to existing methods. Our results show that our approach outperforms MAV3D in terms of quality and accuracy, and provides more diverse and creative results. We also provide a detailed analysis of the user study results, which demonstrate that our method is preferred by users for its high-quality images and realistic motion.
In conclusion, our article presents a novel approach to generating high-quality 4D scenes from text prompts using a hybrid score distillation sampling procedure. By leveraging the strengths of different generative models, we can create 4D scenes that are not only visually realistic but also capture the intended motion and structure. Our approach has important implications for a wide range of applications, including entertainment, education, and virtual reality.