Explore Transformer Architectures

Background: Evaluating the performance of different sequence modeling architectures is a continuing goal in sequential data prediction tasks.

Question / Future Work: Explore self-attention-based Transformer architectures as an alternative to Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) models, as the attention mechanism may be advantageous for capturing relevant temporal dependencies that are not strictly local in the seeing time series.