Time series data is everywhere, from finance to healthcare, and annotating it can be a challenge. To overcome this limitation, researchers have turned to self-supervised learning, which allows them to use unlabeled data without annotations. One approach that has shown promise is contrastive learning, which uses a loss function to train models to learn representations from unlabeled data. In this article, we explore how contrastive learning can be used in time series analysis and provide a new interpretation of the soft contrastive losses.
Semi-supervised Learning
Semi-supervised learning is a technique that combines labeled and unlabeled data to train models. In contrastive learning, the model is trained using unlabeled data, and the loss function is designed to encourage the model to learn representations that are similar for instances that are similar in some way, such as time of day or weather. This allows the model to learn from a large amount of unlabeled data, which can be easier to obtain than labeled data.
Soft Contrastive Losses
The contrastive loss function is a measure of how well the model’s predictions match the true labels. In self-supervised learning, the model is trained using two different views of the same data, such as two different time series of the same stock price. The loss function measures the difference between the model’s predictions for each view and the true labels.
Interpreting Soft Contrastive Losses
Inspired by the fact that the contrastive loss can be interpreted as the cross-entropy loss with virtual labels defined per batch, we define a softmax probability of the relative similarity out of all similarities considered when computing the loss, and interpret our soft contrastive losses as a weighted sum of the cross-entropy losses. This allows us to see the proposed contrastive loss as a scaled KL divergence of the predicted softmax probabilities from the normalized soft assignments.
Hard Assignment
When hard assignment is applied, the loss becomes the standard contrastive loss, which is often called InfoNCE (Oord et al., 2018). This allows us to interpret our soft contrastive losses as a weighted sum of the cross-entropy losses, where the scale is the sum of soft assignments.
Semi-supervised Classification
To evaluate the performance of our semi-supervised learning approach, we fine-tune self- and semi-supervised models on several time series datasets with different labels. The results show that our approach outperforms the self-supervised model in most cases, especially when only 1% of the labels are available.
Conclusion
In this article, we provided a detailed overview of contrastive learning and its application to time series analysis. We also proposed a new interpretation of soft contrastive losses, which allows us to see them as a weighted sum of cross-entropy losses. Our experimental results demonstrate the effectiveness of our approach in semi-supervised classification tasks, outperforming self-supervised models in most cases. By leveraging large amounts of unlabeled data, our approach has the potential to improve the performance of time series analysis tasks, making it easier and more efficient to analyze complex time series data.
