So I want to train LSTM sequence to sequence model, autoencoder, for anomaly detection. The idea is to train it on normal samples and when anomaly comes into model it will not be able to reconstruct it correctly and will have high reconstruction error. I am thinking about how to make the model better, does it make sense to use attention mechanism after encoder network ?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ The answer to that can be highly dataset dependent, and may be hard for others to answer in general. What kind of data do you have? $\endgroup$– Jon NordbyNov 16, 2020 at 14:04
-
$\begingroup$ I have CIC-IDS 2017 data: unb.ca/cic/datasets/ids-2017.html $\endgroup$– pikachuNov 16, 2020 at 14:15
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
I think you can use feature attention.
Firstly you transform the states(n states each with d dimension) from the encoder into a fixed-shape k by d(or any dimension) matrix. In decoding your attention mechanism just pay attention to that matrix, which can not only overcome the information bottleneck of LSTM seq2seq but also speed up the attention.
You'd better not employ the normal attention technique because the reconstruction error would be always very low because it learns to only pay attention to the input in the corresponding position.
answered Nov 22, 2020 at 8:19
-
$\begingroup$ Thanks for the response! If you would have a bit time, could you please add some mathematical explanation why the reconstruction error would be low with normal attention, but not with feature attention? (a beginner on attention here) thanks :) $\endgroup$– pikachuNov 24, 2020 at 9:11
-
$\begingroup$ ... and could you maybe please suggest how to implement this in python ? $\endgroup$– pikachuNov 24, 2020 at 11:58
-
$\begingroup$ @pikachu It's straightforward because it would be just like you train the BERT like transformer with no mask. $\endgroup$ Nov 24, 2020 at 12:14
-
$\begingroup$ @pikachu With normal attention, the network would become an identity function or a null function. Please refer to this article. $\endgroup$ Mar 15, 2021 at 14:54