I am currently trying to get the hang of BERT and Transformers, so I worked through the Paper "Attention Is All You Need". Now I have a hard time understanding how the Key-, Value-, and Query-Matrices for the attention mechanism are obtained.

The paper itself states that:

all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder.

In addition, this tutorial taught me that the inputs (i.e. the outputs from the previous layer) are transformed into $K$, $V$, and $Q$ through three separate matrix multiplications - I found that nowhere in the paper, but it makes sense.

With Multi-Head-Attention, I understand that the inputs are each mapped into several low-dimensional representations. My question now is: In a Multi-Head-Attention-Layer, are there first three weight matrices that map the entire input into $K$, $V$ and $Q$, and then subsequently the transformations into the lower dimensions for each head? Or can the first step be omitted and the inputs be directly transformed into the multi-head-representations?

Edit to specify my question:

In the paper the multi-head-attention is defined with the following formula: $ head_i = Attention(QW_i^Q, KW_i^K, VW_i^V) $.
In this context, are $K$, $V$ and $Q$ identical matrices (the previous layer's output), or are they the result of three separate linear transformations on the previous layer's output?


1 Answer 1


Both questions are actually answered by the same line in the paper, from §3.2.2 “Multi-Head Attention”.

…we found it beneficial to linearly project the queries, keys and values $h$ times with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv -dimensional output values.

That’s your “three separate matrix multiplications”.

There’s no “subsequent” mapping going on, because it wouldn’t help. The composition of two linear mappings (the product of two matrices) is another linear mapping, so it wouldn’t increase the expressive power of the model. You could instead just replace those two parameter matrices with their product.

Bringing this back to your main question: the matrix multiplications directly give you the transformation into what you call the “multi-head-representations” (the inputs to $h$ softmax functions).

  • $\begingroup$ I think we are talking about different things here. My question is: before the linear projections in the lower dimensions, is the input as a whole projected? I added this to the question $\endgroup$
    – Josef
    Commented May 11, 2022 at 15:51
  • $\begingroup$ We are talking about the same thing. If they were 'projected', then there would be no mathematical value to that projection, because they get 'projected' again in the immediate next operation using $W_Q$, $W_K$, and $W_V$—a standard linear algebra result. $\endgroup$ Commented May 11, 2022 at 16:13
  • $\begingroup$ In the question, you ask whether K, Q, and V are identical. In the encoder, yes. The authors write, "The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder." $\endgroup$ Commented May 11, 2022 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.