I have a dataset. Each data point of this set contains a variable length of sequences with 7 letters. For example, Data point 1 has a sequence (A, A, B, E, B, C, D, E, D.....). I used LSTM to predict the next letter from this dataset. I know that adding Attention mechanism can improve the prediction accuracy. I searched for the background of how actually attention mechanism works. I came across various models which work well on the encoder-decoder problem. I am wondering in my case (i.e., predicting the next action/word) how Attention mechanism can improve the accuracy. If anyone can give some intuition or reference, it will be a great benefit for me.
Asked
Viewed
53 times
0
$\begingroup$
$\endgroup$
-
$\begingroup$ if you've read up on attention mechanisms then what exactly is your question here? $\endgroup$ – shimao Apr 11 at 19:34
-
$\begingroup$ Actually, I didn't get good intuition how it works. Particularly, how it can improve the prediction accuracy. So I am looking for some mathematical reference or explanation $\endgroup$ – Bloodstone Programmer Apr 11 at 21:14
-
$\begingroup$ When I learned the attention mechanism I read this paper: Effective Approaches to Attention-based Neural Machine Translation. I think you can utilize the global attention mechnism in your case. $\endgroup$ – Lerner Zhang Apr 25 at 4:30
-
$\begingroup$ What is the task? Classification generating one single letter or generating a sequence which is a follow-up of a sequence you already have? $\endgroup$ – Jindřich Jun 3 at 15:30
