My understanding of attention is that a weighted combination of a set of vectors is somehow combined with the decoder's hidden state. How exactly is it combined? Is it added to the hidden state before it enters the cell at each time step?

In particular, I'm using LuongAttention, an AttentionWrapper, and a GRUCell.


Let me start with describing the general attention mechanism, because it seems important:

  • On a current decoder step $t$, the decoder output is combined from $S+1$ cell states: all $S$ encoder states $(\bar{h}_1, ..., \bar{h}_S)$ and the current decoder state $h_t$.
  • These $S+1$ tensors are squashed into a discrete probability distribution over $S$ values $(p_1, ..., p_S)$, usually by applying softmax to a set of scores $score_s$ for $1 \leq s \leq S$.
  • Here comes the score function: it is a way to produce a number given $h_t$ and $\bar{h}_s$: $score_s = score(h_t, \bar{h}_s)$. It is the first time the decoder state is participating in the attention, for all encoder states.

    Note that there can be different choices of the score function and this is exactly where the choice of wrapper matters: Bahdanau score, Luong score, etc.

  • The result distribution $(p_1, ..., p_S)$ allows to compute a weighted average of encoder states $\bar{h}_1, ..., \bar{h}_S$ to a get a single context vector (tensor to be exact) $c_t$.
  • Finally, the answer to your main question: the context vector $c_t$ is combined with the decoder state $h_t$ to produce an attention vector like this (the second time $h_t$ is used):

$$a_t = tanh(W_c[c_t;h_t])$$

This means the both vectors are stacked and go through one more linear layer with $tanh$ non-linearity. Stacking seems reasonable, because in general $h_t$ and $\bar{h}_s$ (and thus $c_t$) can have different dimensions, so you can't add them easily.

All of this can be summarized in one picture:



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.