Diverse Behaviors through Conditioning: CTT Outperforms SOTA with Semantic Meanings

In this article, we propose a new approach to multimodal decoding called Scene Modes, which is particularly useful for scene-centric models. The traditional method of decoding involves jointly decoding all agents in the scene, which becomes computationally complex as the number of agents increases. To address this issue, we introduce the concept of Scene Modes, where each mode corresponds to one decoding process. This allows us to reduce the computational complexity while maintaining interpretability and expressiveness.
To define Scene Modes, we use a flexible definition that can be adjusted based on the needs of the scene. We also propose key design elements of CTT (Compositional Traffic Theory), which provides a detailed explanation of how Scene Modes work in practice.
The idea behind Scene Modes is to divide the scene into smaller, more manageable parts called "modes." Each mode corresponds to one decoding process, allowing us to reduce the computational complexity without losing important details. Think of it like a recipe book with multiple chapters, each chapter representing a different mode. By choosing the right chapter (or mode), we can simplify the recipe while still achieving the desired outcome.
We also propose using traffic models to provide domain-specific knowledge to LLMs (Large Language Models), which are great at logical reasoning and common-sense knowledge but lack fine manipulation skills when applied to grasping and require an "expert" policy to fill this gap. By combining LLMs with traffic models, we can create a more comprehensive and accurate AI system for autonomous vehicles.
In summary, Scene Modes are a new approach to multimodal decoding that simplifies the decoding process while maintaining interpretability and expressiveness. By dividing the scene into smaller parts called modes, we can reduce computational complexity without losing important details. This approach has many potential applications in autonomous vehicles and other fields where multimodal decoding is necessary.