Adaptive Mixtures of Local Experts: A New Approach to Neural Networks

In this article, we explore a new technique called SwitchHead that improves the attention mechanism used in Transformer models. The traditional attention mechanism is based on computing the weighted sum of input elements using a single set of learnable parameters. However, this can result in redundant computations and limit the model’s expressive power.
To address these limitations, SwitchHead introduces a novel attention layer that splits the computation into two parts: (1) computing the attention weights for each position separately, and (2) combining them to form the final attention matrix. This allows the model to selectively focus on different parts of the input sequence, resulting in improved performance and reduced redundancy.
The authors evaluate SwitchHead on several natural language processing tasks and compare it to traditional attention mechanisms. They find that SwitchHead consistently outperforms the baselines for each dataset and model size considered, except in the case of a large 259M parameter model on the C4 dataset. Additionally, they show that SwitchHead can be used with other attention methods, such as RoPE positional encodings, without any significant loss in performance.
The authors also analyze the attention maps generated by SwitchHead and find that they are qualitatively similar to those produced by dense baselines, indicating a significant reduction in redundancy without a loss of expressivity. They also observe that expert selections are often interpretable, which can be useful for understanding how the model is making its predictions.
Overall, SwitchHead represents a significant improvement over traditional attention mechanisms and demonstrates the potential to improve the performance of Transformer models on natural language processing tasks. By selectively focusing on different parts of the input sequence, SwitchHead can reduce computational complexity while maintaining accuracy, making it an attractive choice for applications where efficiency is crucial.