Large language models (LLMs) have become a crucial component in various natural language processing tasks, with billions of parameters being trained to achieve impressive performance. However, these large models also come with significant computational requirements, which can be challenging for scaling up and deploying them efficiently. In this article, we provide an algorithmic survey of efficient LLMs, focusing on the various approaches that have been proposed to reduce their computational complexity while preserving their accuracy.
Efficient Approaches
- Pruning: One straightforward approach to reducing the computational complexity of LLMs is pruning, which involves removing redundant or unnecessary neurons and connections in the model. This technique has been shown to be effective in various works [267, 268], leading to significant reductions in parameters and computations without compromising performance.
- Low-Rank Parameter-Efficient Fine-Tuning: Another approach is to use low-rank parameter-efficient fine-tuning, which involves training the model with a reduced rank matrix. This technique can be used in combination with pruning to further reduce the computational complexity of the model [319].
- Attention Free Transformer: Some recent works have proposed attention-free transformer models, which eliminate the computationally expensive self-attention mechanism without sacrificing performance. These models have shown promising results in various natural language processing tasks [318].
- Distilling: Another approach is to use distillation techniques, which involve training a smaller model to mimic the behavior of a larger, pre-trained model. This can be useful for reducing the computational complexity of the smaller model without compromising accuracy [120].
The Efficiency Spectrum of Large Language Models
Efficient LLMs are not just about reducing computational complexity; they also need to maintain their accuracy levels. In other words, there is a trade-off between efficiency and accuracy. As shown in Figure 1, the efficiency of LLMs can be divided into four categories based on their computational complexity:
| Efficiency Category | Computational Complexity | Description |
| — | — |
| Big Models | High | These are the largest models with billions of parameters. While they are highly accurate, they also require significant computational resources. Examples include BERT and RoBERTa. |
| Medium Models | Medium | These models have a few hundred million parameters and offer a balance between accuracy and efficiency. Examples include DistilBERT and ALBERT. |
| Small Models | Low | These models have tens of millions of parameters and are the most efficient option for deploying LLMs in resource-constrained environments. Examples include Transformer-XL and ELECTRA. |
| Extremely Small Models | Very Low | These models have a few million parameters and offer extreme efficiency but may compromise on accuracy. Examples include TinyBERT and Simplified BERT. |
Figure 1: The Efficiency Spectrum of Large Language Models
Conclusion
Efficient LLMs have emerged as a crucial area of research in natural language processing, with various approaches proposed to reduce their computational complexity while preserving accuracy. From pruning and low-rank parameter-efficient fine-tuning to attention-free transformer models and distillation techniques, these efficient algorithms can help scale up LLMs for real-world applications. However, the trade-off between efficiency and accuracy remains a challenging area of research, with each model category offering its unique advantages and disadvantages. As the field continues to evolve, we can expect even more sophisticated approaches to emerge in the coming years.
