Improving Generalization in Deep Neural Networks through Low-Rank Approximations

In the field of artificial intelligence, deep neural networks (DNNs) have shown great potential for learning and improving on various tasks. However, one major challenge they face is catastrophic forgetting, where a model forgets previously learned knowledge as it adapts to new tasks. This issue becomes more pronounced when the task distribution shifts significantly over time, causing the model to lose its ability to perform well on both old and new tasks.
To overcome this challenge, researchers have proposed various methods, including rehearsal-based techniques (Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Shmelkov, Schmid, & Alahari, 2017; Chaudhry et al., 2018b, 2021) and architecture-based methods (Mallya & Lazebnik, 2018; Serra et al., 2018). These techniques aim to either prevent the forgetting of previously learned knowledge or enable the model to adapt quickly to new tasks without sacrificing its ability to perform well on old tasks.
One proposed method is hard attention, which involves assigning higher weights to samples that are closer in feature space to the decision boundary. This allows the model to focus more on the samples it has seen before and less on the new ones, reducing the catastrophic forgetting problem (Ye & Bors, 2022). Another method is DRO (Deng et al., 2021), which uses a buffer to store examples from previous tasks and re-samples them with a different probability before each iteration. This helps the model to retain the knowledge from the previous tasks while adapting to new ones.
Another line of research involves using online learning methods, such as ODDL (Wang et al., 2022), which uses a dynamic linear layer to learn the similarity between old and new tasks, and DPCL (Koh et al., 2022), which uses a parameter to control the trade-off between forgetting and adapting. These methods aim to strike a balance between preventing catastrophic forgetting and enabling quick adaptation to new tasks.
In summary, overcoming catastrophic forgetting in deep neural networks is an active research area, with various proposed methods aimed at reducing or eliminating this challenge. These methods include rehearsal-based techniques, architecture-based methods, online learning methods, and hard attention-based approaches. By selecting the appropriate method depending on the specific task and dataset, these methods can help improve the performance of DNNs in various applications while preventing catastrophic forgetting.