So I want to train LSTM sequence to sequence model, autoencoder, for anomaly detection. The idea is to train it on normal samples and when anomaly comes into model it will not be able to reconstruct it correctly and will have high reconstruction error. I am thinking about how to make the model better, does it make sense to use attention mechanism after encoder network ?

  • $\begingroup$ The answer to that can be highly dataset dependent, and may be hard for others to answer in general. What kind of data do you have? $\endgroup$
    – Jon Nordby
    Commented Nov 16, 2020 at 14:04
  • $\begingroup$ I have CIC-IDS 2017 data: unb.ca/cic/datasets/ids-2017.html $\endgroup$
    – pikachu
    Commented Nov 16, 2020 at 14:15

1 Answer 1


I think you can use feature attention.

Firstly you transform the states(n states each with d dimension) from the encoder into a fixed-shape k by d(or any dimension) matrix. In decoding your attention mechanism just pay attention to that matrix, which can not only overcome the information bottleneck of LSTM seq2seq but also speed up the attention.

You'd better not employ the normal attention technique because the reconstruction error would be always very low because it learns to only pay attention to the input in the corresponding position.

Inference: Linformer: Self-Attention with Linear Complexity

  • $\begingroup$ Thanks for the response! If you would have a bit time, could you please add some mathematical explanation why the reconstruction error would be low with normal attention, but not with feature attention? (a beginner on attention here) thanks :) $\endgroup$
    – pikachu
    Commented Nov 24, 2020 at 9:11
  • $\begingroup$ ... and could you maybe please suggest how to implement this in python ? $\endgroup$
    – pikachu
    Commented Nov 24, 2020 at 11:58
  • $\begingroup$ @pikachu It's straightforward because it would be just like you train the BERT like transformer with no mask. $\endgroup$ Commented Nov 24, 2020 at 12:14
  • $\begingroup$ @pikachu With normal attention, the network would become an identity function or a null function. Please refer to this article. $\endgroup$ Commented Mar 15, 2021 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.