In the Attention is all you need paper, the self-attention layer is defined as $\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V$.

I would like to know why a more symmetric design with regards to those 3 matrices isn't favored.

For example, the design could have been made more symmetric with a 4th matrix:

$\text{AttentionAlt}(Q, K, V, W) = \frac{1}{\sqrt{d_k}} \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)VW^T$

This aspect strikes me because about the only structural element (putting the masks aside) in the whole transformer architecture which isn't symmetric with regard to encoder and decoder blocks.

Note: The matrices are explained with an Information Retrieval (IR) analogy: $Q$ is the Query, $K$ is the Key and $V$ is the value. However, this analogy is relatively weak: $Q$ and $K$ are completely symmetrical with one another which is not the case with IR. In the encoder and decoder block, the multi-head attention can be seen as a way to loosely introduce symmetries for those three matrices. However, in the encoder-decoder block, $Q$ and $K$ are connected to the encoder block, while $V$ is connected to the decoder block.

  • $\begingroup$ I think the scaling constant needs to go inside the softmax so that the output probabilities are reasonably scaled. In your scenario, you might vanishing the gradients. Ultimately the output matrix, Z, computed by concatenating all the attention output Values matrices, V, is multiplied by a new dense layer, W. So I'm not sure the design is really all that different from what you propose. $\endgroup$ Commented Aug 22, 2021 at 12:25
  • $\begingroup$ @Learningstatsbyexample Thanks! Edited the example formula. I am not sure either! I suspect the key answer is: we tried, it's slower and not more accurate; but there might be stuff more profound too. $\endgroup$ Commented Aug 23, 2021 at 6:42

1 Answer 1


This is already partially answered in the comment. With the standard attention, you basically multiply $V$ by a vector (or matrix) of probabilities $A$, so that you pay more (higher probabilities) or less (lower probabilities) "attention" to particular values of $V$

$$ \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V = A V $$

What you seem to suggest is to have something like

$$ \frac{1}{\sqrt{d_k}} A VW^T $$

in such a case, you not only attend but also normalize $\tfrac{1}{\sqrt{d_k}}$ and multiply $V$ by additional weights $W^T$. Sure, you can do all kinds of algebraic operations inside a neural network, and such changes in many cases lead to creating new model architectures, but your suggestions go well beyond "just" attending to the values of $V$.

Moreover, in the first case, you only normalize the attention part with regard to the dimensionality $d_k$ of the key $K$, while in the second version you normalize the output in terms of the dimensionality of the key. Those operations have completely different effects, why would the output directly depend on the dimensionality of the key? In the first case, the result is just a convex combination of values $V$ with weights such as $\sum A = 1$. In the second case, you have no guarantees that the output would still be in the range of the values $V$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.