Transformers Efficient and Affordable: Learned Token Pruning, Quantization, and Distillation

In this paper, Xin et al. propose DeeBERT, a technique to accelerate BERT inference by applying dynamic early exiting. BERT (Bidirectional Encoder Representations from Transformers) is a powerful language model trained on large datasets, but its inference time can be slow due to the complexity of the model.
To address this issue, DeeBERT introduces the concept of early exiting, which involves dynamically switching between different parts of the BERT model during inference. By only using the most relevant parts of the model, DeeBERT can significantly reduce the inference time without sacrificing accuracy.
The authors propose two techniques to implement early exiting in DeeBERT: (1) dynamic routing, which selects the most relevant parts of the BERT model based on the input sentence, and (2) hierarchical attention, which uses a hierarchical structure of attention mechanisms to focus on different levels of abstraction in the input data.
DeeBERT is evaluated on several benchmark datasets and shows significant speedups compared to full BERT inference, with minimal loss in accuracy. The authors also demonstrate the versatility of DeeBERT by applying it to various NLP tasks such as question answering, sentiment analysis, and named entity recognition.
In summary, DeeBERT is a powerful technique for accelerating BERT inference without sacrificing accuracy. By dynamically switching between different parts of the model, DeeBERT can significantly reduce the inference time while maintaining the performance of full BERT. This technique has important implications for real-world applications of BERT where speed and efficiency are crucial.