Why is Bahdanau's attention sometimes called concat attention?

Question

I am learning the intuition behind the attention mechanism from

• https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
• https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html

and there is something that I don't quite get. Both entries make reference to some concatenation happening in the decoding stage. From reading Bahdanau's paper, nowhere states that the alignment score is based on the concatenation of the decoder state ($s_i$) and the hidden state ($h_t$). In Luong's paper, this is referred to as the concat attention (the word score is used, though)

$$ \text{score}(h_t; \bar{h}_{s}) = v_a^T \tanh (W_a [h_t; \bar{h}_{s}] ) $$

or in Bahdanau's notation:

$$ a(s_{i−1}, h_j) = v_a^T \tanh (W_a [s_{i−1}; h_{j}] ) $$

In Bahdanau's paper, the alignment score is defined as

$$ a(s_{i−1}, h_j) = v_a^T \tanh (W_a s_{i−1} + U_ah_{j} ) $$

And the only concatenation happening is that of the forward and backward hidden states in the bidirectional encoder. It seems like Vaswani's definition of additive attention makes more sense.

Where is this idea of concatenation coming from?

---

Possibly related

• Is it true that Bahdanau's attention mechanism is not Global like Luong's?

Answer 1

The semicolon operator in the formulas actually denotes concatenation and it the concatenation that they refer to in the paper (opposed to dot product).

The summation in Bahdanau's formulation with the sum of two projections is equivalent to a projection of the vector concatenation ($\oplus$ denotes concatenation):

$$W_a s_{i-j} + U_a h_j = (W_a \oplus U_a) \cdot (s_{i-1} \oplus h_j)$$

It directly follows from the definition of matrix multiplication. Let's call the the dimension of the intermediate projection, $d_s$ dimension of the decoder state, $d_h$ dimension of the encoder state. Then for the $k$-th position in the output:

$$\left[(W_a \oplus U_a) \cdot (s_{i-1} \oplus h_j)\right]_k = \sum_{l=1}^{d_s + d_h} (W_a \oplus U_a)_{k,l} \cdot (s_{i-1} \oplus h_j)_l = \\ \sum_{l=1}^{d_s} (W_a)_{k,l} \cdot (s_{i-1})_l + \sum_{l=1}^{d_h} (U_a)_{k,l} \cdot (h_l)_l = \left[ W_a s_{i-1} \right]_k + \left[ U_a h_j \right]_k $$

So, in the Luong's notation, $W_a \equiv W_a \oplus U_a$ in Bahdanau's notation.