Deep Double Descent for Time Series Forecasting: Avoiding Undertrained Models

In recent years, there has been significant progress in the field of deep learning, particularly with the development of transformer models. However, these models are not universally applicable and their performance can be improved by tailoring them to specific tasks and domains. This article explores the concept of training schemata in other domains beyond natural language processing (NLP) and computer vision, where similar strategies have led to groundbreaking advancements.
The study focused on three well-known transformer models in the field – FEDformer, Autoformer, and Informer – which were found to exhibit different performance improvements when subjected to scaling laws. The research demonstrates that increasing the size of models, data, and computational resources can significantly improve model performance, pushing the boundaries of what machine learning (ML) models can achieve.
To better understand these findings, it’s helpful to compare the field of NLP and computer vision with other domains where similar strategies have been employed. For instance, in the realm of autonomous driving, researchers have combined model architecture innovations with a masking training schema on the model targets to improve performance [19]. Similarly, InstructGPT utilizes reinforcement learning from human feedback (RLHF) to fine-tune existing GPT models, allowing it to follow user instructions more accurately [23].
In essence, these strategies aim to optimize the training schema of deep learning models by leveraging vast amounts of unstructured data to learn general features and representations that can be fine-tuned for specific tasks. By replicating this approach in other domains, there is potential for significant improvements in performance and accuracy.
While this study provides valuable insights into the optimization of transformer models, it’s important to recognize that there are limitations to the findings. The research focused on three well-known models in the field, which may not constitute an exhaustive examination of all available time series models. Nonetheless, the study serves as a starting point for further exploration and evaluation across various applications and contexts.
In conclusion, the article demonstrates that transformer models can be significantly improved by tailoring them to specific tasks and domains through various strategies such as scaling laws, pre-training on vast amounts of unstructured data, and combining model architecture innovations with a masking training schema. These findings have the potential to revolutionize the field of deep learning and enable more accurate and efficient ML models in various applications.