0
$\begingroup$

I just got into models beyond biLSTM, would like to start with applying attention to my existing network (RNN). I find examples for attention always with encoder decoder architecture, however is it possible to use attention without encoder decoder? Is there any example, blog, any kind of content out there discussing an encoder/decoder-free attention model? Or are encoder/decoder simply 100% necessary for the attention mechanism to be applied?

I specifically work with text data on sequence modelling.

$\endgroup$
  • $\begingroup$ You might want to look at 'Attention Is All You Need' paper and Transformer Networks. $\endgroup$ – Jakub Bartczuk Nov 15 '18 at 9:44
  • $\begingroup$ I did, however I understood the paper as the implementation was not with an RNN but some kind of different network entirely, which is why they were able to implement attention without encoder-decoder. $\endgroup$ – xyz Nov 15 '18 at 9:54
  • $\begingroup$ I just want to add, regarding the Attention Is All You Need paper - here is an annotated version of the paper - and it indeed does make use of an encoder and decoder. nlp.seas.harvard.edu/2018/04/03/attention.html $\endgroup$ – xyz Nov 28 '18 at 20:51
0
$\begingroup$

Whether a model has an encoder and decoder is orthogonal to whether it uses attention. Actually I doubt the term encoder-decoder has any well defined meaning, as you can take any model, cut it in half any way you like, and then call the first half the encoder and the second half the decoder.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.