I am learning the intuition behind the attention mechanism from

and there is something that I don't quite get. Both entries make reference to some concatenation happening in the decoding stage. From reading Bahdanau's paper, nowhere states that the alignment score is based on the concatenation of the decoder state ($s_i$) and the hidden state ($h_t$). In Luong's paper, this is referred to as the concat attention (the word score is used, though)

$$ \text{score}(h_t; \bar{h}_{s}) = v_a^T \tanh (W_a [h_t; \bar{h}_{s}] ) $$

or in Bahdanau's notation:

$$ a(s_{i−1}, h_j) = v_a^T \tanh (W_a [s_{i−1}; h_{j}] ) $$

In Bahdanau's paper, the alignment score is defined as

$$ a(s_{i−1}, h_j) = v_a^T \tanh (W_a s_{i−1} + U_ah_{j} ) $$

And the only concatenation happening is that of the forward and backward hidden states in the bidirectional encoder. It seems like Vaswani's definition of additive attention makes more sense.

Where is this idea of concatenation coming from?

Possibly related


1 Answer 1


The semicolon operator in the formulas actually denotes concatenation and it the concatenation that they refer to in the paper (opposed to dot product).

The summation in Bahdanau's formulation with the sum of two projections is equivalent to a projection of the vector concatenation ($\oplus$ denotes concatenation):

$$W_a s_{i-j} + U_a h_j = (W_a \oplus U_a) \cdot (s_{i-1} \oplus h_j)$$

It directly follows from the definition of matrix multiplication. Let's call the the dimension of the intermediate projection, $d_s$ dimension of the decoder state, $d_h$ dimension of the encoder state. Then for the $k$-th position in the output:

$$\left[(W_a \oplus U_a) \cdot (s_{i-1} \oplus h_j)\right]_k = \sum_{l=1}^{d_s + d_h} (W_a \oplus U_a)_{k,l} \cdot (s_{i-1} \oplus h_j)_l = \\ \sum_{l=1}^{d_s} (W_a)_{k,l} \cdot (s_{i-1})_l + \sum_{l=1}^{d_h} (U_a)_{k,l} \cdot (h_l)_l = \left[ W_a s_{i-1} \right]_k + \left[ U_a h_j \right]_k $$

So, in the Luong's notation, $W_a \equiv W_a \oplus U_a$ in Bahdanau's notation.

  • $\begingroup$ Hey, thanks! I had to write down a simple example with letters to follow your notation, but I think I get it now. $\endgroup$
    – bluesmonk
    Commented May 18, 2021 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.