In this research paper, the authors aim to address a critical challenge in machine learning called "catastrophic forgetting," which occurs when a model learns new knowledge at the expense of previously acquired knowledge. This problem is particularly pronounced in federated learning, where multiple models are trained simultaneously on different devices, and data is shared between them.
To overcome this challenge, the authors propose a novel approach called "SSRE" (Structural Reorganization with Main Branch Knowledge Distillation), which combines two strategies:
- Main Branch Knowledge Distillation: This involves training a smaller model (called the "main branch") on the entire dataset, while transferring the knowledge from the larger model (called the "side branch") to the main branch through a process called "knowledge distillation." This helps to preserve the previously acquired knowledge and prevent catastrophic forgetting.
- Structural Reorganization: The authors propose a novel reorganization strategy, called "structural expansion," which integrates the knowledge from the side branch into the main branch while retaining the old knowledge. This is achieved by adding new layers to the main branch and gradually merging them with the existing layers.
The proposed approach is evaluated on several benchmark datasets and shows significant improvement in terms of preserving previously acquired knowledge while learning new tasks. The authors also provide an analysis of the impact of different hyperparameters on the performance of their approach.
In summary, the authors propose a novel approach called SSRE to address the challenge of catastrophic forgetting in federated learning. By combining main branch knowledge distillation and structural reorganization, they are able to preserve previously acquired knowledge while learning new tasks, resulting in better overall performance.
