I have a dataset. Each data point of this set contains a variable length of sequences with 7 letters. For example, Data point 1 has a sequence (A, A, B, E, B, C, D, E, D.....). I used LSTM to predict the next letter from this dataset. I know that adding Attention mechanism can improve the prediction accuracy. I searched for the background of how actually attention mechanism works. I came across various models which work well on the encoder-decoder problem. I am wondering in my case (i.e., predicting the next action/word) how Attention mechanism can improve the accuracy. If anyone can give some intuition or reference, it will be a great benefit for me.

  • $\begingroup$ if you've read up on attention mechanisms then what exactly is your question here? $\endgroup$ – shimao Apr 11 at 19:34
  • $\begingroup$ Actually, I didn't get good intuition how it works. Particularly, how it can improve the prediction accuracy. So I am looking for some mathematical reference or explanation $\endgroup$ – Bloodstone Programmer Apr 11 at 21:14
  • $\begingroup$ When I learned the attention mechanism I read this paper: Effective Approaches to Attention-based Neural Machine Translation. I think you can utilize the global attention mechnism in your case. $\endgroup$ – Lerner Zhang Apr 25 at 4:30
  • $\begingroup$ What is the task? Classification generating one single letter or generating a sequence which is a follow-up of a sequence you already have? $\endgroup$ – Jindřich Jun 3 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.