Exploring Efficient Deep Neural Networks with Sparsity and Group Spatiality for Multi-Task Learning

In the field of Machine Learning (ML), there is a principle known as parsimony, which suggests that simpler models are preferred over more complex ones when both models can fit the data equally well. This principle is particularly important in Multi-task Learning (MTL), where models are trained on multiple tasks simultaneously. However, in recent years, the concept of over-parameterization has become intertwined with MTL, where models have more parameters than necessary to fit their training data.
To overcome these issues, researchers have been exploring the idea of model sparsity, which involves removing unnecessary features or parameters from a model to make it simpler and more interpretable. This approach can lead to increased interpretability, reduced overfitting, efficient computation, and the ability to identify the most informative features, resulting in an efficient learning process.
The authors of this article review recent studies on parameter efficiency in MTL and explore the literature on structured sparsity in general contexts and specifically within MTL. They argue that while over-parameterization can enable networks to approximate complex mappings and simplify the loss function, making optimization easier, model sparsification can provide significant benefits by removing unnecessary features or parameters.
To illustrate this concept, the authors use an analogy of a kitchen where chefs have different ingredients at their disposal. Just like how too many ingredients can make a dish difficult to prepare and less enjoyable, over-parameterization in MTL can lead to complex models that are hard to interpret and optimize. On the other hand, using only the essential ingredients, like a simple recipe with few ingredients, can result in a delicious and easy-to-prepare dish, similar to model sparsity in MTL.
In summary, the article highlights the importance of model sparsity in MTL, where simpler models are preferred over more complex ones when both models can fit the data equally well. By removing unnecessary features or parameters, model sparsity can lead to increased interpretability, reduced overfitting, efficient computation, and the ability to identify the most informative features, resulting in an efficient learning process.