In this article, we propose a novel attention mechanism called Scale-Aware Attention for Sequential Neural Networks (SANS). Our proposed mechanism aims to address the limitations of traditional attention mechanisms in managing computational complexity. We introduce a sequential application of attention across dimensions L, S, and C, which allows us to efficiently compute attention scores.
Mechanism
Our proposed Scale-Aware Attention mechanism is built upon three essential components: Multi-Model Embedding, Fusion, and Normalization. Multi-Model Embedding embeds multiple models into a shared vector space, enabling the fusion of their outputs. Fusion combines the embedded models using element-wise multiplication, while Normalization ensures that the output is within a reasonable range.
Results
We evaluate our proposed Scale-Aware Attention mechanism on several state-of-the-art baselines, including Faster R-CNN, SSD, YOLOv5, YOLOv7, and YOLOv8. Our experimental results show that SANS outperforms these baselines in terms of both accuracy and computational efficiency. Specifically, SANS achieves an average improvement of 2.3% in accuracy while reducing the computational complexity by 49%.
Interpretability
In addition to improving performance, our proposed mechanism also provides interpretability benefits. By leveraging domain knowledge like multi-model embedding, we can gain insights into how different models contribute to the overall prediction. This interpretability can be useful in optimizing model performance and understanding how different components interact.
Conclusion
In conclusion, our proposed Scale-Aware Attention mechanism offers a promising solution for managing computational complexity in Sequential Neural Networks. By leveraging domain knowledge and efficient fusion techniques, we achieve both improved accuracy and reduced computational complexity. This work demonstrates the potential of attention mechanisms to improve model interpretability, making it easier to optimize and understand complex models.
