I have data in the shape (batch, seq_len, features) that is a time series sliding window. In essence, I'm using the most recent seq_len steps in order to predict a single target variable. This means that the output of the last seq_len value in my MultiHeadAttention layer should be the predicted value.

I've made many attempts at generating different attention_masks to use in Keras' MultiHeadAttention but none of them quite capture the behavior I want, inevitably leading to poor results. Ultimately I only want the importance of each seq_len query step relative to the last key step. It's basically an autoregressive additive model using the transformer architecture (only using the encoder). The last step is to tf.reduce_sum over the entire seq_len in order to get the output.

Future modifications to the attention layer might be using teacher forcing which should further improve the learning phase and reduce the obvious influence of the correlation of the last value with itself, but I can't figure out how to correctly mask in the first place for continuous time series data like this. To be clear this is NOT an NLP model.

  • $\begingroup$ You might look at the paper that introduces Longformer, which discusses a sliding window attention. $\endgroup$ Commented May 12, 2022 at 19:14
  • $\begingroup$ @AryaMcCarthy that doesn't appear to be quite what I need. I would need something that attends all of query's seq_len to the last of key's seq_len. $\endgroup$ Commented May 12, 2022 at 20:55

1 Answer 1


In the attention model, the mask required is of the shape (batch, queries, keys) so in order to train the entire horizon (queries) on the last value (last key):

def timeseries_sliding_window(length):
    """Returns a mask of shape (length, length) where only the last entry of the 
    2nd dimension is of relevance to all elements of the 1st
    return tf.concat([tf.zeros((length - 1, length)), tf.ones((1, length))], 0)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.