Transformers vs RNNs in Spatiotemporal Data Imputation: A Comparative Study

In this article, we delve into the realm of time series imputation, specifically focusing on methods that utilize attention mechanisms. We begin by discussing the challenges associated with imputing missing values in time series data and how traditional recurrent neural network (RNN) models struggle to address these issues. We then introduce attention-based methods, which were first introduced by Vaswani et al. (2017) and have since shown remarkable performance in various natural language processing tasks. These methods offer an alternative to RNNs and are better at capturing long-term dependencies and understanding global context, making them particularly useful for accurate imputation in time series data with complex relationships.
We explore several attention-based models, including the Spatial Attention Transformer (SAITS), which leverages self-attention mechanisms to fill in missing values in multivariate time series. Tashiro et al. (2021) proposed a unique architecture of 2D attention, consisting of temporal attention and feature attention, for imputing missing values in multivariate time series. These models demonstrate remarkable performance compared to RNNs, as they avoid error propagation, are easier to optimize, and offer a more robust framework for handling sparse observations.
However, one limitation of existing transformer-based imputation models is their quadratic complexity with respect to the input length, making them unscalable for large spatiotemporal datasets. To address this issue, we propose an enhancement that employs multiple layers of the encoder, providing the model with more opportunities to learn abstract representations of the input data. The final output from the encoder is formed by concatenating the outputs from all spatiotemporal attention layers and processing them through a generic MLP to generate the final imputation.
To further illustrate the effectiveness of these models, we conduct a case study on soil moisture imputation using both remote-sensed and ground-based datasets. Our results demonstrate that attention-based methods outperform traditional RNNs in terms of accuracy and computational efficiency, making them a promising approach for solving time series imputation problems.
In summary, this article provides a comprehensive overview of time series imputation using attention mechanisms, highlighting their advantages over traditional RNNs and discussing several state-of-the-art models that have shown remarkable performance in various applications. By leveraging self-attention mechanisms, these models can capture long-term dependencies and understand global context more effectively, making them particularly useful for accurate imputation in time series data with complex relationships. However, the quadratic complexity of these models remains a limitation, which we address by proposing an enhancement that employs multiple layers of the encoder. Our case study on soil moisture imputation demonstrates the effectiveness of attention-based methods in improving accuracy and computational efficiency compared to traditional RNNs.