Low-Rank Neural Networks for Continual Learning

Continual learning is a vital aspect of deep neural network (DNN) training, as it enables models to adapt to new tasks while retaining previous knowledge. However, this process can lead to overparameterization, where the model has more neurons than necessary for the task at hand. To address this issue, researchers propose selective disabling of neurons, which involves identifying and removing unnecessary neurons during training.
The authors present a novel approach called "lottery ticket hypothesis," which suggests that sub-networks with fewer neurons can match or surpass the performance of larger networks. By leveraging this insight, the authors develop a method to identify critical neurons for task performance and disable non-critical ones during training. This ensures that low-rank feature filters (ff) can be learned after training without compromising performance.
The proposed approach consists of two stages: pretraining and fine-tuning. In the pretraining stage, a sub-network with fewer neurons per layer is obtained by selecting only the most critical neurons based on their importance for the task at hand. During fine-tuning, the non-critical neurons are disabled, allowing the model to adapt to new tasks without overwriting previously learned knowledge.
The authors demonstrate the effectiveness of their approach through experiments on several benchmark datasets. The results show that selective disabling of neurons leads to improved performance in continual learning scenarios while reducing the capacity of the model.
In conclusion, selective disabling of neurons is a promising technique for improving continual learning in DNNs. By leveraging the lottery ticket hypothesis, this approach identifies and removes unnecessary neurons during training, enabling the model to adapt to new tasks while retaining previous knowledge. This demystifies complex concepts by using analogies to help readers understand the importance of selective disabling of neurons in continual learning.