I understand the use of attention mechanisms in the encoder-decoder for sequence-to-sequence problem such as a language translator.

I am just trying to figure out whether it is possible to use attention mechanisms with standard auto-encoders for feature extraction where the goal is to compress the data into a latent vector?

Suppose we had a time series data with N dimensions and we wanted to use an auto-encoder with attention mechanisms (I am thinking of a self-attention because I think it is more appropriate in this case - I might be wrong) to better learn interdependence among the input sequence and thus we would get a better latent vector L.

Or it could be better to use Recurrent Neural Network or its variants in this case.

Does anyone have better thoughts or an intuition behind this?

  • 1
    $\begingroup$ Technically, RNN does "compress the data into a latent vector" as well, just using different approach. $\endgroup$
    – Tim
    Oct 23 '20 at 13:24
  • $\begingroup$ Thanks, @Tim. Yes, I am aware of this. What I am asking is that would it be possible (in theory) to use attention mechanisms with an autoencoder for feature extraction to enforce encoder part to learn better representation and may get better more richer latent vector without losing much useful information. I am just trying to see whether there is a potential to add attention in this case or not. I did not find much resources that discuss this in particular. $\endgroup$
    – Amhs_11
    Oct 23 '20 at 13:33
  • 4
    $\begingroup$ This is an interesting question, and I'm curious to know the answer. But since transformers are relatively new, I wonder if an answer is known yet -- it's possible that no one's researched this exact question and published their findings yet. $\endgroup$
    – Sycorax
    Oct 26 '20 at 1:38
  • $\begingroup$ @Sycorax, thanks for your comment. Yes, I totally agree with you. Perhaps, some intuition or thought on this would be appreciated. At least, we can see some high level discussions to better understand it. $\endgroup$
    – Amhs_11
    Oct 26 '20 at 2:16
  • $\begingroup$ I don't see the difference between your scenario and seq2seq. Also, you can use both RNN encoders/decoders and attention at the same time. $\endgroup$
    – Firebug
    Oct 31 '20 at 17:13

I think attention can help. Please refer to this answer.

There are many ways for you to incorporate the attention with an autoencoder. The simplest way is just to borrow the idea from BERT but make the middle layers thinner.

  • $\begingroup$ Thank you for your answer, I definitely should check BERT. Thank you!. Just one thing, I just went through your answer that you provided in the link. If I understand correctly, you suggest to place the attention on the decoder part?. My question how would this help to better extract features in latent space. $\endgroup$
    – Amhs_11
    Nov 22 '20 at 23:25
  • $\begingroup$ @Amhs_11 No, on the states from the encoder. $\endgroup$ Nov 23 '20 at 13:00
  • $\begingroup$ Thanks @Lerner Zhang, okay then I guess it would be added to the encoder part. I still do not get it because you mentioned that "In decoding your attention mechanism just pay attention to that matrix" in the linked answer. Anyway, thank you very much for your clarification. $\endgroup$
    – Amhs_11
    Nov 23 '20 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.