Study on Attention-based LSTM Model for Multivariate Time-series Prediction

Abstract

Given previous observations of a multivariate time-series, how can we accurately predict the future value of several steps ahead? With the continuous development of sensor systems and computer systems, time-series prediction techniques are playing more and more important roles in various fields, such as finance, energy, and traffics. Many models have been proposed for time-series prediction tasks, such as Autoregressive model, Vector Autoregressive model, and Recurrent Neural Networks (RNNs). However, these models still have limitations like failure in modeling non-linearity and long-term dependencies in time-series. Among all the proposed approaches, the Temporal Pattern Attention (TPA), which is an attention-based LSTM model, achieves state-of-the-art performance on several real-world multivariate time-series datasets. In this thesis, we study three factors that effect the prediction performance of TPA model, which are the Recurrent Neural Network RNN layer, the attention mechanism, and the Convolutional Neural Network for temporal patter detection. For recurrent layer, we implement bi-directional LSTMs that can extract information from the input sequence in both forward and backward directions. In addition, we design two attention mechanisms, each of which assigns attention weights in different directions. We study the effect of both attention mechanisms on TPA model. Finally, to validate the Convolutional Neural Network (CNN) for temporal pattern detection, we implement a TPA model without CNN. We test all of these factors using several real-world time-series datasets from different fields. The experimental results indicate the validity of these factors.